Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [X......X.........X..................X.......XXX......X......] duration: 52.314593883s trace: https://jaeger.dev.storj.io/trace/7e25338f73046b2a FAIL (2025-12-26 10:30:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033130872 (498h55m22.904213918s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033471341 (498h55m22.904554386s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033892292 (498h55m22.904975328s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034387946 (498h55m22.905470991s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034807164 (498h55m22.905890209s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035259836 (498h55m22.906342881s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035727828 (498h55m22.906810863s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036033871 (498h55m22.907116906s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036579810 (498h55m22.907662855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.037767882 (498h55m22.908850927s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.038064076 (498h55m22.909147122s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.039224646 (498h55m22.910307691s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040040249 (498h55m22.911123294s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040366420 (498h55m22.911449455s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.041569130 (498h55m22.912652175s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:24.513239450 (498h55m24.384322485s) [upload=437904]: Next returned nil segment Dec 26 10:24:24.513254640 (498h55m24.384337685s) [upload=437904]: Waiting for error group managing segments... Dec 26 10:24:24.926029601 (498h55m24.797112636s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.027458408 (498h55m24.898541443s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.094888666 (498h55m24.965971701s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.102988388 (498h55m24.974071433s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.109420056 (498h55m24.980503091s) [noise=true , address=77.90.31.15:4732 , piece_id=QDQXC2LEVGS7K5OHDX7Q3ZETJJ37XFXIS6BXXNFCUTKFEMDLJ23A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.122147140 (498h55m24.993230186s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.187517306 (498h55m25.058600351s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.195537778 (498h55m25.066620823s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.255475810 (498h55m25.126558855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.546862739 (498h55m25.417945784s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.792060605 (498h55m25.66314365s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.809945911 (498h55m25.681028946s) [noise=true , address=145.79.227.44:4221 , piece_id=IZLW4HA64OMYW23ZBPBZU45GR4QCEREPUL76QL6RWWUA3LKKFN2A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.860843739 (498h55m25.731926774s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861204966 (498h55m25.732288001s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861232289 (498h55m25.732315324s) [seg_pos={0 0}, upload=437904]: Segment reached optimal threshold of 14 pieces. Dec 26 10:24:25.861471414 (498h55m25.732554449s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861479399 (498h55m25.732562434s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861580682 (498h55m25.732663717s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection Dec 26 10:24:25.861589008 (498h55m25.732672043s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861840307 (498h55m25.732923342s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861847050 (498h55m25.732930085s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861929887 (498h55m25.733012932s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861939285 (498h55m25.73302232s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861959854 (498h55m25.733042889s) [seg_pos={0 0}, upload=437904]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:24:25.862033124 (498h55m25.733116159s) [upload=437904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:24:25.862037923 (498h55m25.733120958s) [upload=437904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:24:26.151364428 (498h55m26.022447463s) [upload=437904]: Done uploading segments... Dec 26 10:24:26.151393914 (498h55m26.022476949s) [upload=437904]: Upload finished. err: <nil> Duration: 52.314622036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X.....X] duration: 30.003712556s trace: https://jaeger.dev.storj.io/trace/13587ab9c29ac635 FAIL (2025-12-26 10:30:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F, Piece ID: KZSRCM6YQVB2YLDHGONZE6PTPJNVPPLORTYVWSXV7PJOIXXZ2K3Q) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7CRDZJ4BPYSNVMDTBY6LZ2TXYLLVKZG3JWLCBKXHYN3YRGMFUNYA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: APGQRIUQXQPHM6MJ4WEDQS2YVRIUOB3B2WL7STY53BZC73DFHRVA) context deadline exceeded ======LOGS====== Dec 26 10:30:12.681859623 (499h1m12.552942658s) [upload=437980]: Starting upload Dec 26 10:30:12.681930027 (499h1m12.553013062s) [upload=437980]: Done starting upload Dec 26 10:30:12.682013387 (499h1m12.553096432s) [upload=437980]: Uploading segments... Dec 26 10:30:12.682127454 (499h1m12.553210499s) [upload=437980]: Got next segment. Inline: false Dec 26 10:30:12.682165997 (499h1m12.553249042s) [upload=437980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:30:12.682170596 (499h1m12.553253631s) [upload=437980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:30:12.924920241 (499h1m12.796003296s) [seg_pos={0 0}, upload=437980]: Begin upload segment... Dec 26 10:30:12.924972882 (499h1m12.796055927s) [seg_pos={0 0}, upload=437980]: Done begin upload segment. Dec 26 10:30:12.925004733 (499h1m12.796087778s) [noise=true , address=45.91.116.6:4974 , piece_id=7CRDZJ4BPYSNVMDTBY6LZ2TXYLLVKZG3JWLCBKXHYN3YRGMFUNYA, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.926823055 (499h1m12.7979061s) [noise=true , address=45.91.116.10:4084 , piece_id=LN2PFBABMQKGYY6FHFZKFE7E4N32JD3J2MXSSUIOZ3QNQVPREB7Q, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.926857961 (499h1m12.797941007s) [noise=true , address=145.79.227.11:4730 , piece_id=K6I3LTE2YTS4UWL5O6WVUA36LL3WQTLBDAOJ3RCLEW3VVDA3WFIA, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.927408229 (499h1m12.798491275s) [noise=true , address=77.90.31.17:4770 , piece_id=SQTD2CJW5HKC5HR4EU4AZAE42HWDGWFUBYQRZWLXIDRZI6MAWQHQ, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.927709203 (499h1m12.798792248s) [noise=true , address=45.91.116.8:4045 , piece_id=HKGUK33JHQMUPJJTPAFM3UN6667OJKCBPGGEHZ4I3F6DPU7HCX4Q, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.927998704 (499h1m12.799081749s) [noise=true , address=45.91.116.4:4939 , piece_id=JGOL6ZKZLZNKKO57CVPAAGJMEXYFFIVTQB3ZQBLUU4MWSDQINKAA, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.928142588 (499h1m12.799225643s) [noise=true , address=145.79.227.51:4410 , piece_id=JORQEKIMIKOEQDTSTNT3VBNPLVQUL7SB56T3EC67ZKXWYJADZBRQ, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.928541767 (499h1m12.799624802s) [noise=true , address=145.79.227.47:4267 , piece_id=GXD6LBSELZA42GE36HPPTIN6RM2THN7DNXGE3JLIKBEBKMGY4JAA, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.928657128 (499h1m12.799740163s) [noise=true , address=45.91.116.9:4262 , piece_id=UXCQTESNECGK46LQCPGADR4AEHAUSAGRICGVHSRRTK3PTEGUXMSQ, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.928933654 (499h1m12.800016699s) [noise=true , address=45.91.116.5:4298 , piece_id=APGQRIUQXQPHM6MJ4WEDQS2YVRIUOB3B2WL7STY53BZC73DFHRVA, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.928986885 (499h1m12.80006996s) [noise=true , address=145.79.227.50:4374 , piece_id=RJ67Q3DJSNCXV2M2QOWUIYTT73G45J4TCQSRDKKRRSVY77TXJA5A, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.929364625 (499h1m12.80044767s) [noise=true , address=145.79.227.37:4566 , piece_id=KZSRCM6YQVB2YLDHGONZE6PTPJNVPPLORTYVWSXV7PJOIXXZ2K3Q, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.929666380 (499h1m12.800749415s) [noise=true , address=45.91.116.13:4153 , piece_id=YKYSRQJD3N4WAC3GHMDWCIMBBOF3FKTX6HVXFYD5X2PAMPLELSPQ, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.930059708 (499h1m12.801142753s) [noise=true , address=45.91.116.11:4120 , piece_id=ZUUQL72DY2Z5D3EEH3BHYOIL7QKDIVKNEHK2L7ZRM3VN5TE3OWBA, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.930649030 (499h1m12.801732065s) [noise=true , address=145.79.227.35:4482 , piece_id=5GYC4BRZA675TMB3VZHLOD4KYGYKRT2WY4M3SKWGIKMGYVWUVCXQ, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:12.931640328 (499h1m12.802723373s) [noise=true , address=145.79.227.46:4603 , piece_id=IYE5JFRSO6M7IMQ5VZ6UYP6MFP7E4GIVQWKBNTC57UX7NWWXJDAA, seg_pos={0 0}, upload=437980]: Uploading piece... Dec 26 10:30:14.641266715 (499h1m14.51234977s) [upload=437980]: Next returned nil segment Dec 26 10:30:14.641283457 (499h1m14.512366502s) [upload=437980]: Waiting for error group managing segments... Dec 26 10:30:14.924123611 (499h1m14.795206646s) [noise=true , address=145.79.227.37:4566 , piece_id=KZSRCM6YQVB2YLDHGONZE6PTPJNVPPLORTYVWSXV7PJOIXXZ2K3Q, seg_pos={0 0}, upload=437980]: Done uploading piece. err: <nil> Dec 26 10:30:14.933190676 (499h1m14.804273711s) [noise=true , address=145.79.227.50:4374 , piece_id=RJ67Q3DJSNCXV2M2QOWUIYTT73G45J4TCQSRDKKRRSVY77TXJA5A, seg_pos={0 0}, upload=437980]: Done uploading piece. err: <nil> Dec 26 10:30:14.933317978 (499h1m14.804401013s) [noise=true , address=145.79.227.11:4730 , piece_id=K6I3LTE2YTS4UWL5O6WVUA36LL3WQTLBDAOJ3RCLEW3VVDA3WFIA, seg_pos={0 0}, upload=437980]: Done uploading piece. err: <nil> Dec 26 10:30:14.939112482 (499h1m14.810195527s) [noise=true , address=145.79.227.35:4482 , piece_id=5GYC4BRZA675TMB3VZHLOD4KYGYKRT2WY4M3SKWGIKMGYVWUVCXQ, seg_pos={0 0}, upload=437980]: Done uploading piece. err: <nil> Dec 26 10:30:14.950977786 (499h1m14.822060821s) [noise=true , address=145.79.227.47:4267 , piece_id=GXD6LBSELZA42GE36HPPTIN6RM2THN7DNXGE3JLIKBEBKMGY4JAA, seg_pos={0 0}, upload=437980]: Done uploading piece. err: <nil> Dec 26 10:30:14.956792448 (499h1m14.827875493s) [noise=true , address=145.79.227.51:4410 , piece_id=JORQEKIMIKOEQDTSTNT3VBNPLVQUL7SB56T3EC67ZKXWYJADZBRQ, seg_pos={0 0}, upload=437980]: Done uploading piece. err: <nil> Dec 26 10:30:14.959815294 (499h1m14.830898329s) [noise=true , address=145.79.227.46:4603 , piece_id=IYE5JFRSO6M7IMQ5VZ6UYP6MFP7E4GIVQWKBNTC57UX7NWWXJDAA, seg_pos={0 0}, upload=437980]: Done uploading piece. err: <nil> Dec 26 10:30:15.195997783 (499h1m15.067080829s) [noise=true , address=45.91.116.6:4974 , piece_id=7CRDZJ4BPYSNVMDTBY6LZ2TXYLLVKZG3JWLCBKXHYN3YRGMFUNYA, seg_pos={0 0}, upload=437980]: Done uploading piece. err: <nil> Dec 26 10:30:15.296288815 (499h1m15.16737186s) [noise=true , address=45.91.116.8:4045 , piece_id=HKGUK33JHQMUPJJTPAFM3UN6667OJKCBPGGEHZ4I3F6DPU7HCX4Q, seg_pos={0 0}, upload=437980]: Done uploading piece. err: <nil> Dec 26 10:30:15.297533335 (499h1m15.16861638s) [noise=true , address=45.91.116.11:4120 , piece_id=ZUUQL72DY2Z5D3EEH3BHYOIL7QKDIVKNEHK2L7ZRM3VN5TE3OWBA, seg_pos={0 0}, upload=437980]: Done uploading piece. err: <nil> Dec 26 10:30:15.314163491 (499h1m15.185246536s) [noise=true , address=45.91.116.4:4939 , piece_id=JGOL6ZKZLZNKKO57CVPAAGJMEXYFFIVTQB3ZQBLUU4MWSDQINKAA, seg_pos={0 0}, upload=437980]: Done uploading piece. err: <nil> Dec 26 10:30:15.590048411 (499h1m15.461131456s) [noise=true , address=45.91.116.5:4298 , piece_id=APGQRIUQXQPHM6MJ4WEDQS2YVRIUOB3B2WL7STY53BZC73DFHRVA, seg_pos={0 0}, upload=437980]: Done uploading piece. err: <nil> Dec 26 10:30:15.965149324 (499h1m15.836232359s) [noise=true , address=45.91.116.9:4262 , piece_id=UXCQTESNECGK46LQCPGADR4AEHAUSAGRICGVHSRRTK3PTEGUXMSQ, seg_pos={0 0}, upload=437980]: Done uploading piece. err: <nil> Dec 26 10:30:15.965172588 (499h1m15.836255633s) [seg_pos={0 0}, upload=437980]: Segment reached optimal threshold of 13 pieces. Dec 26 10:30:15.965352591 (499h1m15.836435636s) [noise=true , address=45.91.116.13:4153 , piece_id=YKYSRQJD3N4WAC3GHMDWCIMBBOF3FKTX6HVXFYD5X2PAMPLELSPQ, seg_pos={0 0}, upload=437980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 10:30:15.965358562 (499h1m15.836441597s) [noise=true , address=45.91.116.13:4153 , piece_id=YKYSRQJD3N4WAC3GHMDWCIMBBOF3FKTX6HVXFYD5X2PAMPLELSPQ, seg_pos={0 0}, upload=437980]: Long tail cancellation due to optimal threshold Dec 26 10:30:15.965512786 (499h1m15.836595831s) [noise=true , address=77.90.31.17:4770 , piece_id=SQTD2CJW5HKC5HR4EU4AZAE42HWDGWFUBYQRZWLXIDRZI6MAWQHQ, seg_pos={0 0}, upload=437980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 10:30:15.965529598 (499h1m15.836612633s) [noise=true , address=77.90.31.17:4770 , piece_id=SQTD2CJW5HKC5HR4EU4AZAE42HWDGWFUBYQRZWLXIDRZI6MAWQHQ, seg_pos={0 0}, upload=437980]: Long tail cancellation due to optimal threshold Dec 26 10:30:15.965550548 (499h1m15.836633593s) [noise=true , address=45.91.116.10:4084 , piece_id=LN2PFBABMQKGYY6FHFZKFE7E4N32JD3J2MXSSUIOZ3QNQVPREB7Q, seg_pos={0 0}, upload=437980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 10:30:15.965580054 (499h1m15.836663099s) [noise=true , address=45.91.116.10:4084 , piece_id=LN2PFBABMQKGYY6FHFZKFE7E4N32JD3J2MXSSUIOZ3QNQVPREB7Q, seg_pos={0 0}, upload=437980]: Long tail cancellation due to optimal threshold Dec 26 10:30:15.965623607 (499h1m15.836706652s) [seg_pos={0 0}, upload=437980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:30:15.965691306 (499h1m15.836774341s) [upload=437980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:30:15.965697337 (499h1m15.836780392s) [upload=437980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:30:16.256842355 (499h1m16.1279254s) [upload=437980]: Done uploading segments... Dec 26 10:30:16.256861711 (499h1m16.127944746s) [upload=437980]: Upload finished. err: <nil> Duration: 30.003738155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X.....] duration: 6.036266115s trace: https://jaeger.dev.storj.io/trace/5acd663927bec960 FAIL (2025-12-26 10:29:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ) context deadline exceeded ======LOGS====== 0:25:28.105454108 (498h56m27.976537143s) [upload=437919]: Starting upload Dec 26 10:25:28.105529822 (498h56m27.976612867s) [upload=437919]: Done starting upload Dec 26 10:25:28.105615174 (498h56m27.97669822s) [upload=437919]: Uploading segments... Dec 26 10:25:28.105756253 (498h56m27.976839298s) [upload=437919]: Got next segment. Inline: false Dec 26 10:25:28.105787723 (498h56m27.976870768s) [upload=437919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:25:28.105792612 (498h56m27.976875677s) [upload=437919]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:25:28.347683773 (498h56m28.218766808s) [seg_pos={0 0}, upload=437919]: Begin upload segment... Dec 26 10:25:28.347735281 (498h56m28.218818326s) [seg_pos={0 0}, upload=437919]: Done begin upload segment. Dec 26 10:25:28.347768394 (498h56m28.218851429s) [noise=true , address=77.90.31.17:4773 , piece_id=U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.347775288 (498h56m28.218858323s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348182993 (498h56m28.219266038s) [noise=true , address=145.79.227.4:4185 , piece_id=UXP7AP26EF2S44AHCVTKILCT7QZS6PKXJN4G5NMNWE4V4KA7X6TQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348188744 (498h56m28.219271779s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348476423 (498h56m28.219559468s) [noise=true , address=45.91.116.4:4937 , piece_id=BYNEKIDVTUEDJ3D63FF3NMJ2MNDR7BVK3PV37PM43YYMYVCXERUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348544993 (498h56m28.219628038s) [noise=true , address=145.79.227.19:4078 , piece_id=CK267Q7KZNDZHQOOCASBK4T22XW2FB3C2OHEXA3HVJVRADKKMVPQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348775251 (498h56m28.219858296s) [noise=true , address=145.79.227.37:4566 , piece_id=OFJQA2IMHUIQERT4VO5IDJGOTS6NDQZQPX5SGPQ52HWLR6QK6C2Q, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349056907 (498h56m28.220139942s) [noise=true , address=77.90.31.18:4808 , piece_id=DT4NSARC4O2YK2X7ZIOXTWNJDLC53WXOQ6T5L7SUJQEK7UUYFUUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349346079 (498h56m28.220429124s) [noise=true , address=145.79.227.5:4222 , piece_id=BC3KZHAPQM2FX6ORY2DFTRFXVS5YXIQA5P5ZHI5MFTOY2MDU7WWA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349638154 (498h56m28.220721189s) [noise=true , address=77.90.31.16:4736 , piece_id=FUBKPFM2VTHHXX6JQZJRIA3LCS4A3663YJI4GKLK5BT7GF2KZYCA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350014451 (498h56m28.221097486s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350396899 (498h56m28.221479944s) [noise=true , address=45.91.116.9:4262 , piece_id=XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350785810 (498h56m28.221868845s) [noise=true , address=45.91.116.6:4974 , piece_id=TAMZQNHVRWTYHZCYNUMO7HNBMFFETLVQQVMGHC5SSKNHLB7ABGDA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.351265854 (498h56m28.222348889s) [noise=true , address=77.90.31.14:4663 , piece_id=NJJZPZDTPCJDKSSH3UMMQCXJ7ETMMG44WNUZPJHLXOPCWLVFCDAQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.351746580 (498h56m28.222829615s) [noise=true , address=145.79.227.29:4942 , piece_id=3E3AAJLKD42KSYXLC7JWTFHGLXKYDAHHZLKZWCY2HHSCZ7MVMAUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.353707453 (498h56m28.224790498s) [noise=true , address=145.79.227.43:4157 , piece_id=XLBHS7BTQOUTHXYMMLCQRLGJAUSTBPXTVLMX7ZNPOYLIWDWTKAJQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:30.072964459 (498h56m29.944047514s) [upload=437919]: Next returned nil segment Dec 26 10:25:30.073021316 (498h56m29.944104361s) [upload=437919]: Waiting for error group managing segments... Dec 26 10:25:30.360224859 (498h56m30.231307894s) [noise=true , address=145.79.227.19:4078 , piece_id=CK267Q7KZNDZHQOOCASBK4T22XW2FB3C2OHEXA3HVJVRADKKMVPQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.369668272 (498h56m30.240751307s) [noise=true , address=145.79.227.37:4566 , piece_id=OFJQA2IMHUIQERT4VO5IDJGOTS6NDQZQPX5SGPQ52HWLR6QK6C2Q, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.376976877 (498h56m30.248059912s) [noise=true , address=145.79.227.43:4157 , piece_id=XLBHS7BTQOUTHXYMMLCQRLGJAUSTBPXTVLMX7ZNPOYLIWDWTKAJQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.379469584 (498h56m30.250552619s) [noise=true , address=145.79.227.4:4185 , piece_id=UXP7AP26EF2S44AHCVTKILCT7QZS6PKXJN4G5NMNWE4V4KA7X6TQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.380055119 (498h56m30.251138165s) [noise=true , address=145.79.227.5:4222 , piece_id=BC3KZHAPQM2FX6ORY2DFTRFXVS5YXIQA5P5ZHI5MFTOY2MDU7WWA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.388429615 (498h56m30.25951265s) [noise=true , address=145.79.227.29:4942 , piece_id=3E3AAJLKD42KSYXLC7JWTFHGLXKYDAHHZLKZWCY2HHSCZ7MVMAUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.560310524 (498h56m30.431393559s) [noise=true , address=77.90.31.17:4773 , piece_id=U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.612021641 (498h56m30.483104676s) [noise=true , address=45.91.116.4:4937 , piece_id=BYNEKIDVTUEDJ3D63FF3NMJ2MNDR7BVK3PV37PM43YYMYVCXERUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.631168459 (498h56m30.502251494s) [noise=true , address=45.91.116.9:4262 , piece_id=XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.025679662 (498h56m30.896762707s) [noise=true , address=77.90.31.18:4808 , piece_id=DT4NSARC4O2YK2X7ZIOXTWNJDLC53WXOQ6T5L7SUJQEK7UUYFUUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.376755608 (498h56m31.247838643s) [noise=true , address=45.91.116.6:4974 , piece_id=TAMZQNHVRWTYHZCYNUMO7HNBMFFETLVQQVMGHC5SSKNHLB7ABGDA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.546990753 (498h56m31.418073788s) [noise=true , address=77.90.31.14:4663 , piece_id=NJJZPZDTPCJDKSSH3UMMQCXJ7ETMMG44WNUZPJHLXOPCWLVFCDAQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.709881213 (498h56m31.580964258s) [noise=true , address=77.90.31.16:4736 , piece_id=FUBKPFM2VTHHXX6JQZJRIA3LCS4A3663YJI4GKLK5BT7GF2KZYCA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.709902273 (498h56m31.580985308s) [seg_pos={0 0}, upload=437919]: Segment reached optimal threshold of 13 pieces. Dec 26 10:25:31.710110780 (498h56m31.581193825s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 10:25:31.710117674 (498h56m31.581200719s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710461367 (498h56m31.581544413s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 26 10:25:31.710479342 (498h56m31.581562387s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710598328 (498h56m31.581681363s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection Dec 26 10:25:31.710606485 (498h56m31.58168952s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710631842 (498h56m31.581714877s) [seg_pos={0 0}, upload=437919]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:25:31.710707096 (498h56m31.581790141s) [upload=437919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:25:31.710712426 (498h56m31.581795461s) [upload=437919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:25:31.998705452 (498h56m31.869788487s) [upload=437919]: Done uploading segments... Dec 26 10:25:31.998722373 (498h56m31.869805408s) [upload=437919]: Upload finished. err: <nil> Duration: 6.036291775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X......X.........X..................X.......XXX......X.....] duration: 46.596460898s trace: https://jaeger.dev.storj.io/trace/7e25338f73046b2a FAIL (2025-12-26 10:29:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033130872 (498h55m22.904213918s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033471341 (498h55m22.904554386s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033892292 (498h55m22.904975328s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034387946 (498h55m22.905470991s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034807164 (498h55m22.905890209s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035259836 (498h55m22.906342881s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035727828 (498h55m22.906810863s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036033871 (498h55m22.907116906s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036579810 (498h55m22.907662855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.037767882 (498h55m22.908850927s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.038064076 (498h55m22.909147122s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.039224646 (498h55m22.910307691s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040040249 (498h55m22.911123294s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040366420 (498h55m22.911449455s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.041569130 (498h55m22.912652175s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:24.513239450 (498h55m24.384322485s) [upload=437904]: Next returned nil segment Dec 26 10:24:24.513254640 (498h55m24.384337685s) [upload=437904]: Waiting for error group managing segments... Dec 26 10:24:24.926029601 (498h55m24.797112636s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.027458408 (498h55m24.898541443s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.094888666 (498h55m24.965971701s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.102988388 (498h55m24.974071433s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.109420056 (498h55m24.980503091s) [noise=true , address=77.90.31.15:4732 , piece_id=QDQXC2LEVGS7K5OHDX7Q3ZETJJ37XFXIS6BXXNFCUTKFEMDLJ23A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.122147140 (498h55m24.993230186s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.187517306 (498h55m25.058600351s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.195537778 (498h55m25.066620823s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.255475810 (498h55m25.126558855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.546862739 (498h55m25.417945784s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.792060605 (498h55m25.66314365s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.809945911 (498h55m25.681028946s) [noise=true , address=145.79.227.44:4221 , piece_id=IZLW4HA64OMYW23ZBPBZU45GR4QCEREPUL76QL6RWWUA3LKKFN2A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.860843739 (498h55m25.731926774s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861204966 (498h55m25.732288001s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861232289 (498h55m25.732315324s) [seg_pos={0 0}, upload=437904]: Segment reached optimal threshold of 14 pieces. Dec 26 10:24:25.861471414 (498h55m25.732554449s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861479399 (498h55m25.732562434s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861580682 (498h55m25.732663717s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection Dec 26 10:24:25.861589008 (498h55m25.732672043s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861840307 (498h55m25.732923342s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861847050 (498h55m25.732930085s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861929887 (498h55m25.733012932s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861939285 (498h55m25.73302232s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861959854 (498h55m25.733042889s) [seg_pos={0 0}, upload=437904]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:24:25.862033124 (498h55m25.733116159s) [upload=437904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:24:25.862037923 (498h55m25.733120958s) [upload=437904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:24:26.151364428 (498h55m26.022447463s) [upload=437904]: Done uploading segments... Dec 26 10:24:26.151393914 (498h55m26.022476949s) [upload=437904]: Upload finished. err: <nil> Duration: 46.596479403s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X....] duration: 5.91270234s trace: https://jaeger.dev.storj.io/trace/5acd663927bec960 FAIL (2025-12-26 10:29:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ) context deadline exceeded ======LOGS====== 0:25:28.105454108 (498h56m27.976537143s) [upload=437919]: Starting upload Dec 26 10:25:28.105529822 (498h56m27.976612867s) [upload=437919]: Done starting upload Dec 26 10:25:28.105615174 (498h56m27.97669822s) [upload=437919]: Uploading segments... Dec 26 10:25:28.105756253 (498h56m27.976839298s) [upload=437919]: Got next segment. Inline: false Dec 26 10:25:28.105787723 (498h56m27.976870768s) [upload=437919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:25:28.105792612 (498h56m27.976875677s) [upload=437919]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:25:28.347683773 (498h56m28.218766808s) [seg_pos={0 0}, upload=437919]: Begin upload segment... Dec 26 10:25:28.347735281 (498h56m28.218818326s) [seg_pos={0 0}, upload=437919]: Done begin upload segment. Dec 26 10:25:28.347768394 (498h56m28.218851429s) [noise=true , address=77.90.31.17:4773 , piece_id=U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.347775288 (498h56m28.218858323s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348182993 (498h56m28.219266038s) [noise=true , address=145.79.227.4:4185 , piece_id=UXP7AP26EF2S44AHCVTKILCT7QZS6PKXJN4G5NMNWE4V4KA7X6TQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348188744 (498h56m28.219271779s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348476423 (498h56m28.219559468s) [noise=true , address=45.91.116.4:4937 , piece_id=BYNEKIDVTUEDJ3D63FF3NMJ2MNDR7BVK3PV37PM43YYMYVCXERUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348544993 (498h56m28.219628038s) [noise=true , address=145.79.227.19:4078 , piece_id=CK267Q7KZNDZHQOOCASBK4T22XW2FB3C2OHEXA3HVJVRADKKMVPQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348775251 (498h56m28.219858296s) [noise=true , address=145.79.227.37:4566 , piece_id=OFJQA2IMHUIQERT4VO5IDJGOTS6NDQZQPX5SGPQ52HWLR6QK6C2Q, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349056907 (498h56m28.220139942s) [noise=true , address=77.90.31.18:4808 , piece_id=DT4NSARC4O2YK2X7ZIOXTWNJDLC53WXOQ6T5L7SUJQEK7UUYFUUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349346079 (498h56m28.220429124s) [noise=true , address=145.79.227.5:4222 , piece_id=BC3KZHAPQM2FX6ORY2DFTRFXVS5YXIQA5P5ZHI5MFTOY2MDU7WWA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349638154 (498h56m28.220721189s) [noise=true , address=77.90.31.16:4736 , piece_id=FUBKPFM2VTHHXX6JQZJRIA3LCS4A3663YJI4GKLK5BT7GF2KZYCA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350014451 (498h56m28.221097486s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350396899 (498h56m28.221479944s) [noise=true , address=45.91.116.9:4262 , piece_id=XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350785810 (498h56m28.221868845s) [noise=true , address=45.91.116.6:4974 , piece_id=TAMZQNHVRWTYHZCYNUMO7HNBMFFETLVQQVMGHC5SSKNHLB7ABGDA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.351265854 (498h56m28.222348889s) [noise=true , address=77.90.31.14:4663 , piece_id=NJJZPZDTPCJDKSSH3UMMQCXJ7ETMMG44WNUZPJHLXOPCWLVFCDAQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.351746580 (498h56m28.222829615s) [noise=true , address=145.79.227.29:4942 , piece_id=3E3AAJLKD42KSYXLC7JWTFHGLXKYDAHHZLKZWCY2HHSCZ7MVMAUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.353707453 (498h56m28.224790498s) [noise=true , address=145.79.227.43:4157 , piece_id=XLBHS7BTQOUTHXYMMLCQRLGJAUSTBPXTVLMX7ZNPOYLIWDWTKAJQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:30.072964459 (498h56m29.944047514s) [upload=437919]: Next returned nil segment Dec 26 10:25:30.073021316 (498h56m29.944104361s) [upload=437919]: Waiting for error group managing segments... Dec 26 10:25:30.360224859 (498h56m30.231307894s) [noise=true , address=145.79.227.19:4078 , piece_id=CK267Q7KZNDZHQOOCASBK4T22XW2FB3C2OHEXA3HVJVRADKKMVPQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.369668272 (498h56m30.240751307s) [noise=true , address=145.79.227.37:4566 , piece_id=OFJQA2IMHUIQERT4VO5IDJGOTS6NDQZQPX5SGPQ52HWLR6QK6C2Q, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.376976877 (498h56m30.248059912s) [noise=true , address=145.79.227.43:4157 , piece_id=XLBHS7BTQOUTHXYMMLCQRLGJAUSTBPXTVLMX7ZNPOYLIWDWTKAJQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.379469584 (498h56m30.250552619s) [noise=true , address=145.79.227.4:4185 , piece_id=UXP7AP26EF2S44AHCVTKILCT7QZS6PKXJN4G5NMNWE4V4KA7X6TQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.380055119 (498h56m30.251138165s) [noise=true , address=145.79.227.5:4222 , piece_id=BC3KZHAPQM2FX6ORY2DFTRFXVS5YXIQA5P5ZHI5MFTOY2MDU7WWA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.388429615 (498h56m30.25951265s) [noise=true , address=145.79.227.29:4942 , piece_id=3E3AAJLKD42KSYXLC7JWTFHGLXKYDAHHZLKZWCY2HHSCZ7MVMAUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.560310524 (498h56m30.431393559s) [noise=true , address=77.90.31.17:4773 , piece_id=U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.612021641 (498h56m30.483104676s) [noise=true , address=45.91.116.4:4937 , piece_id=BYNEKIDVTUEDJ3D63FF3NMJ2MNDR7BVK3PV37PM43YYMYVCXERUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.631168459 (498h56m30.502251494s) [noise=true , address=45.91.116.9:4262 , piece_id=XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.025679662 (498h56m30.896762707s) [noise=true , address=77.90.31.18:4808 , piece_id=DT4NSARC4O2YK2X7ZIOXTWNJDLC53WXOQ6T5L7SUJQEK7UUYFUUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.376755608 (498h56m31.247838643s) [noise=true , address=45.91.116.6:4974 , piece_id=TAMZQNHVRWTYHZCYNUMO7HNBMFFETLVQQVMGHC5SSKNHLB7ABGDA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.546990753 (498h56m31.418073788s) [noise=true , address=77.90.31.14:4663 , piece_id=NJJZPZDTPCJDKSSH3UMMQCXJ7ETMMG44WNUZPJHLXOPCWLVFCDAQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.709881213 (498h56m31.580964258s) [noise=true , address=77.90.31.16:4736 , piece_id=FUBKPFM2VTHHXX6JQZJRIA3LCS4A3663YJI4GKLK5BT7GF2KZYCA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.709902273 (498h56m31.580985308s) [seg_pos={0 0}, upload=437919]: Segment reached optimal threshold of 13 pieces. Dec 26 10:25:31.710110780 (498h56m31.581193825s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 10:25:31.710117674 (498h56m31.581200719s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710461367 (498h56m31.581544413s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 26 10:25:31.710479342 (498h56m31.581562387s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710598328 (498h56m31.581681363s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection Dec 26 10:25:31.710606485 (498h56m31.58168952s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710631842 (498h56m31.581714877s) [seg_pos={0 0}, upload=437919]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:25:31.710707096 (498h56m31.581790141s) [upload=437919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:25:31.710712426 (498h56m31.581795461s) [upload=437919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:25:31.998705452 (498h56m31.869788487s) [upload=437919]: Done uploading segments... Dec 26 10:25:31.998722373 (498h56m31.869805408s) [upload=437919]: Upload finished. err: <nil> Duration: 5.912729102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X......X.........X..................X.......XXX......X....] duration: 45.37347525s trace: https://jaeger.dev.storj.io/trace/7e25338f73046b2a FAIL (2025-12-26 10:28:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033130872 (498h55m22.904213918s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033471341 (498h55m22.904554386s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033892292 (498h55m22.904975328s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034387946 (498h55m22.905470991s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034807164 (498h55m22.905890209s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035259836 (498h55m22.906342881s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035727828 (498h55m22.906810863s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036033871 (498h55m22.907116906s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036579810 (498h55m22.907662855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.037767882 (498h55m22.908850927s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.038064076 (498h55m22.909147122s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.039224646 (498h55m22.910307691s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040040249 (498h55m22.911123294s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040366420 (498h55m22.911449455s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.041569130 (498h55m22.912652175s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:24.513239450 (498h55m24.384322485s) [upload=437904]: Next returned nil segment Dec 26 10:24:24.513254640 (498h55m24.384337685s) [upload=437904]: Waiting for error group managing segments... Dec 26 10:24:24.926029601 (498h55m24.797112636s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.027458408 (498h55m24.898541443s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.094888666 (498h55m24.965971701s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.102988388 (498h55m24.974071433s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.109420056 (498h55m24.980503091s) [noise=true , address=77.90.31.15:4732 , piece_id=QDQXC2LEVGS7K5OHDX7Q3ZETJJ37XFXIS6BXXNFCUTKFEMDLJ23A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.122147140 (498h55m24.993230186s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.187517306 (498h55m25.058600351s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.195537778 (498h55m25.066620823s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.255475810 (498h55m25.126558855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.546862739 (498h55m25.417945784s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.792060605 (498h55m25.66314365s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.809945911 (498h55m25.681028946s) [noise=true , address=145.79.227.44:4221 , piece_id=IZLW4HA64OMYW23ZBPBZU45GR4QCEREPUL76QL6RWWUA3LKKFN2A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.860843739 (498h55m25.731926774s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861204966 (498h55m25.732288001s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861232289 (498h55m25.732315324s) [seg_pos={0 0}, upload=437904]: Segment reached optimal threshold of 14 pieces. Dec 26 10:24:25.861471414 (498h55m25.732554449s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861479399 (498h55m25.732562434s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861580682 (498h55m25.732663717s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection Dec 26 10:24:25.861589008 (498h55m25.732672043s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861840307 (498h55m25.732923342s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861847050 (498h55m25.732930085s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861929887 (498h55m25.733012932s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861939285 (498h55m25.73302232s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861959854 (498h55m25.733042889s) [seg_pos={0 0}, upload=437904]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:24:25.862033124 (498h55m25.733116159s) [upload=437904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:24:25.862037923 (498h55m25.733120958s) [upload=437904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:24:26.151364428 (498h55m26.022447463s) [upload=437904]: Done uploading segments... Dec 26 10:24:26.151393914 (498h55m26.022476949s) [upload=437904]: Upload finished. err: <nil> Duration: 45.373503572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X...] duration: 5.720593911s trace: https://jaeger.dev.storj.io/trace/5acd663927bec960 FAIL (2025-12-26 10:28:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ) context deadline exceeded ======LOGS====== 0:25:28.105454108 (498h56m27.976537143s) [upload=437919]: Starting upload Dec 26 10:25:28.105529822 (498h56m27.976612867s) [upload=437919]: Done starting upload Dec 26 10:25:28.105615174 (498h56m27.97669822s) [upload=437919]: Uploading segments... Dec 26 10:25:28.105756253 (498h56m27.976839298s) [upload=437919]: Got next segment. Inline: false Dec 26 10:25:28.105787723 (498h56m27.976870768s) [upload=437919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:25:28.105792612 (498h56m27.976875677s) [upload=437919]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:25:28.347683773 (498h56m28.218766808s) [seg_pos={0 0}, upload=437919]: Begin upload segment... Dec 26 10:25:28.347735281 (498h56m28.218818326s) [seg_pos={0 0}, upload=437919]: Done begin upload segment. Dec 26 10:25:28.347768394 (498h56m28.218851429s) [noise=true , address=77.90.31.17:4773 , piece_id=U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.347775288 (498h56m28.218858323s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348182993 (498h56m28.219266038s) [noise=true , address=145.79.227.4:4185 , piece_id=UXP7AP26EF2S44AHCVTKILCT7QZS6PKXJN4G5NMNWE4V4KA7X6TQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348188744 (498h56m28.219271779s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348476423 (498h56m28.219559468s) [noise=true , address=45.91.116.4:4937 , piece_id=BYNEKIDVTUEDJ3D63FF3NMJ2MNDR7BVK3PV37PM43YYMYVCXERUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348544993 (498h56m28.219628038s) [noise=true , address=145.79.227.19:4078 , piece_id=CK267Q7KZNDZHQOOCASBK4T22XW2FB3C2OHEXA3HVJVRADKKMVPQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348775251 (498h56m28.219858296s) [noise=true , address=145.79.227.37:4566 , piece_id=OFJQA2IMHUIQERT4VO5IDJGOTS6NDQZQPX5SGPQ52HWLR6QK6C2Q, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349056907 (498h56m28.220139942s) [noise=true , address=77.90.31.18:4808 , piece_id=DT4NSARC4O2YK2X7ZIOXTWNJDLC53WXOQ6T5L7SUJQEK7UUYFUUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349346079 (498h56m28.220429124s) [noise=true , address=145.79.227.5:4222 , piece_id=BC3KZHAPQM2FX6ORY2DFTRFXVS5YXIQA5P5ZHI5MFTOY2MDU7WWA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349638154 (498h56m28.220721189s) [noise=true , address=77.90.31.16:4736 , piece_id=FUBKPFM2VTHHXX6JQZJRIA3LCS4A3663YJI4GKLK5BT7GF2KZYCA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350014451 (498h56m28.221097486s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350396899 (498h56m28.221479944s) [noise=true , address=45.91.116.9:4262 , piece_id=XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350785810 (498h56m28.221868845s) [noise=true , address=45.91.116.6:4974 , piece_id=TAMZQNHVRWTYHZCYNUMO7HNBMFFETLVQQVMGHC5SSKNHLB7ABGDA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.351265854 (498h56m28.222348889s) [noise=true , address=77.90.31.14:4663 , piece_id=NJJZPZDTPCJDKSSH3UMMQCXJ7ETMMG44WNUZPJHLXOPCWLVFCDAQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.351746580 (498h56m28.222829615s) [noise=true , address=145.79.227.29:4942 , piece_id=3E3AAJLKD42KSYXLC7JWTFHGLXKYDAHHZLKZWCY2HHSCZ7MVMAUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.353707453 (498h56m28.224790498s) [noise=true , address=145.79.227.43:4157 , piece_id=XLBHS7BTQOUTHXYMMLCQRLGJAUSTBPXTVLMX7ZNPOYLIWDWTKAJQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:30.072964459 (498h56m29.944047514s) [upload=437919]: Next returned nil segment Dec 26 10:25:30.073021316 (498h56m29.944104361s) [upload=437919]: Waiting for error group managing segments... Dec 26 10:25:30.360224859 (498h56m30.231307894s) [noise=true , address=145.79.227.19:4078 , piece_id=CK267Q7KZNDZHQOOCASBK4T22XW2FB3C2OHEXA3HVJVRADKKMVPQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.369668272 (498h56m30.240751307s) [noise=true , address=145.79.227.37:4566 , piece_id=OFJQA2IMHUIQERT4VO5IDJGOTS6NDQZQPX5SGPQ52HWLR6QK6C2Q, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.376976877 (498h56m30.248059912s) [noise=true , address=145.79.227.43:4157 , piece_id=XLBHS7BTQOUTHXYMMLCQRLGJAUSTBPXTVLMX7ZNPOYLIWDWTKAJQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.379469584 (498h56m30.250552619s) [noise=true , address=145.79.227.4:4185 , piece_id=UXP7AP26EF2S44AHCVTKILCT7QZS6PKXJN4G5NMNWE4V4KA7X6TQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.380055119 (498h56m30.251138165s) [noise=true , address=145.79.227.5:4222 , piece_id=BC3KZHAPQM2FX6ORY2DFTRFXVS5YXIQA5P5ZHI5MFTOY2MDU7WWA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.388429615 (498h56m30.25951265s) [noise=true , address=145.79.227.29:4942 , piece_id=3E3AAJLKD42KSYXLC7JWTFHGLXKYDAHHZLKZWCY2HHSCZ7MVMAUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.560310524 (498h56m30.431393559s) [noise=true , address=77.90.31.17:4773 , piece_id=U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.612021641 (498h56m30.483104676s) [noise=true , address=45.91.116.4:4937 , piece_id=BYNEKIDVTUEDJ3D63FF3NMJ2MNDR7BVK3PV37PM43YYMYVCXERUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.631168459 (498h56m30.502251494s) [noise=true , address=45.91.116.9:4262 , piece_id=XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.025679662 (498h56m30.896762707s) [noise=true , address=77.90.31.18:4808 , piece_id=DT4NSARC4O2YK2X7ZIOXTWNJDLC53WXOQ6T5L7SUJQEK7UUYFUUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.376755608 (498h56m31.247838643s) [noise=true , address=45.91.116.6:4974 , piece_id=TAMZQNHVRWTYHZCYNUMO7HNBMFFETLVQQVMGHC5SSKNHLB7ABGDA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.546990753 (498h56m31.418073788s) [noise=true , address=77.90.31.14:4663 , piece_id=NJJZPZDTPCJDKSSH3UMMQCXJ7ETMMG44WNUZPJHLXOPCWLVFCDAQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.709881213 (498h56m31.580964258s) [noise=true , address=77.90.31.16:4736 , piece_id=FUBKPFM2VTHHXX6JQZJRIA3LCS4A3663YJI4GKLK5BT7GF2KZYCA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.709902273 (498h56m31.580985308s) [seg_pos={0 0}, upload=437919]: Segment reached optimal threshold of 13 pieces. Dec 26 10:25:31.710110780 (498h56m31.581193825s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 10:25:31.710117674 (498h56m31.581200719s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710461367 (498h56m31.581544413s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 26 10:25:31.710479342 (498h56m31.581562387s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710598328 (498h56m31.581681363s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection Dec 26 10:25:31.710606485 (498h56m31.58168952s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710631842 (498h56m31.581714877s) [seg_pos={0 0}, upload=437919]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:25:31.710707096 (498h56m31.581790141s) [upload=437919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:25:31.710712426 (498h56m31.581795461s) [upload=437919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:25:31.998705452 (498h56m31.869788487s) [upload=437919]: Done uploading segments... Dec 26 10:25:31.998722373 (498h56m31.869805408s) [upload=437919]: Upload finished. err: <nil> Duration: 5.720625441s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X......X.........X..................X.......XXX......X...] duration: 38.66956357s trace: https://jaeger.dev.storj.io/trace/7e25338f73046b2a FAIL (2025-12-26 10:27:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033130872 (498h55m22.904213918s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033471341 (498h55m22.904554386s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033892292 (498h55m22.904975328s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034387946 (498h55m22.905470991s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034807164 (498h55m22.905890209s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035259836 (498h55m22.906342881s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035727828 (498h55m22.906810863s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036033871 (498h55m22.907116906s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036579810 (498h55m22.907662855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.037767882 (498h55m22.908850927s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.038064076 (498h55m22.909147122s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.039224646 (498h55m22.910307691s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040040249 (498h55m22.911123294s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040366420 (498h55m22.911449455s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.041569130 (498h55m22.912652175s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:24.513239450 (498h55m24.384322485s) [upload=437904]: Next returned nil segment Dec 26 10:24:24.513254640 (498h55m24.384337685s) [upload=437904]: Waiting for error group managing segments... Dec 26 10:24:24.926029601 (498h55m24.797112636s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.027458408 (498h55m24.898541443s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.094888666 (498h55m24.965971701s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.102988388 (498h55m24.974071433s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.109420056 (498h55m24.980503091s) [noise=true , address=77.90.31.15:4732 , piece_id=QDQXC2LEVGS7K5OHDX7Q3ZETJJ37XFXIS6BXXNFCUTKFEMDLJ23A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.122147140 (498h55m24.993230186s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.187517306 (498h55m25.058600351s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.195537778 (498h55m25.066620823s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.255475810 (498h55m25.126558855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.546862739 (498h55m25.417945784s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.792060605 (498h55m25.66314365s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.809945911 (498h55m25.681028946s) [noise=true , address=145.79.227.44:4221 , piece_id=IZLW4HA64OMYW23ZBPBZU45GR4QCEREPUL76QL6RWWUA3LKKFN2A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.860843739 (498h55m25.731926774s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861204966 (498h55m25.732288001s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861232289 (498h55m25.732315324s) [seg_pos={0 0}, upload=437904]: Segment reached optimal threshold of 14 pieces. Dec 26 10:24:25.861471414 (498h55m25.732554449s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861479399 (498h55m25.732562434s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861580682 (498h55m25.732663717s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection Dec 26 10:24:25.861589008 (498h55m25.732672043s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861840307 (498h55m25.732923342s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861847050 (498h55m25.732930085s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861929887 (498h55m25.733012932s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861939285 (498h55m25.73302232s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861959854 (498h55m25.733042889s) [seg_pos={0 0}, upload=437904]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:24:25.862033124 (498h55m25.733116159s) [upload=437904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:24:25.862037923 (498h55m25.733120958s) [upload=437904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:24:26.151364428 (498h55m26.022447463s) [upload=437904]: Done uploading segments... Dec 26 10:24:26.151393914 (498h55m26.022476949s) [upload=437904]: Upload finished. err: <nil> Duration: 38.669580331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X..] duration: 6.077109288s trace: https://jaeger.dev.storj.io/trace/5acd663927bec960 FAIL (2025-12-26 10:27:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ) context deadline exceeded ======LOGS====== 0:25:28.105454108 (498h56m27.976537143s) [upload=437919]: Starting upload Dec 26 10:25:28.105529822 (498h56m27.976612867s) [upload=437919]: Done starting upload Dec 26 10:25:28.105615174 (498h56m27.97669822s) [upload=437919]: Uploading segments... Dec 26 10:25:28.105756253 (498h56m27.976839298s) [upload=437919]: Got next segment. Inline: false Dec 26 10:25:28.105787723 (498h56m27.976870768s) [upload=437919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:25:28.105792612 (498h56m27.976875677s) [upload=437919]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:25:28.347683773 (498h56m28.218766808s) [seg_pos={0 0}, upload=437919]: Begin upload segment... Dec 26 10:25:28.347735281 (498h56m28.218818326s) [seg_pos={0 0}, upload=437919]: Done begin upload segment. Dec 26 10:25:28.347768394 (498h56m28.218851429s) [noise=true , address=77.90.31.17:4773 , piece_id=U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.347775288 (498h56m28.218858323s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348182993 (498h56m28.219266038s) [noise=true , address=145.79.227.4:4185 , piece_id=UXP7AP26EF2S44AHCVTKILCT7QZS6PKXJN4G5NMNWE4V4KA7X6TQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348188744 (498h56m28.219271779s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348476423 (498h56m28.219559468s) [noise=true , address=45.91.116.4:4937 , piece_id=BYNEKIDVTUEDJ3D63FF3NMJ2MNDR7BVK3PV37PM43YYMYVCXERUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348544993 (498h56m28.219628038s) [noise=true , address=145.79.227.19:4078 , piece_id=CK267Q7KZNDZHQOOCASBK4T22XW2FB3C2OHEXA3HVJVRADKKMVPQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348775251 (498h56m28.219858296s) [noise=true , address=145.79.227.37:4566 , piece_id=OFJQA2IMHUIQERT4VO5IDJGOTS6NDQZQPX5SGPQ52HWLR6QK6C2Q, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349056907 (498h56m28.220139942s) [noise=true , address=77.90.31.18:4808 , piece_id=DT4NSARC4O2YK2X7ZIOXTWNJDLC53WXOQ6T5L7SUJQEK7UUYFUUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349346079 (498h56m28.220429124s) [noise=true , address=145.79.227.5:4222 , piece_id=BC3KZHAPQM2FX6ORY2DFTRFXVS5YXIQA5P5ZHI5MFTOY2MDU7WWA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349638154 (498h56m28.220721189s) [noise=true , address=77.90.31.16:4736 , piece_id=FUBKPFM2VTHHXX6JQZJRIA3LCS4A3663YJI4GKLK5BT7GF2KZYCA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350014451 (498h56m28.221097486s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350396899 (498h56m28.221479944s) [noise=true , address=45.91.116.9:4262 , piece_id=XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350785810 (498h56m28.221868845s) [noise=true , address=45.91.116.6:4974 , piece_id=TAMZQNHVRWTYHZCYNUMO7HNBMFFETLVQQVMGHC5SSKNHLB7ABGDA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.351265854 (498h56m28.222348889s) [noise=true , address=77.90.31.14:4663 , piece_id=NJJZPZDTPCJDKSSH3UMMQCXJ7ETMMG44WNUZPJHLXOPCWLVFCDAQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.351746580 (498h56m28.222829615s) [noise=true , address=145.79.227.29:4942 , piece_id=3E3AAJLKD42KSYXLC7JWTFHGLXKYDAHHZLKZWCY2HHSCZ7MVMAUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.353707453 (498h56m28.224790498s) [noise=true , address=145.79.227.43:4157 , piece_id=XLBHS7BTQOUTHXYMMLCQRLGJAUSTBPXTVLMX7ZNPOYLIWDWTKAJQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:30.072964459 (498h56m29.944047514s) [upload=437919]: Next returned nil segment Dec 26 10:25:30.073021316 (498h56m29.944104361s) [upload=437919]: Waiting for error group managing segments... Dec 26 10:25:30.360224859 (498h56m30.231307894s) [noise=true , address=145.79.227.19:4078 , piece_id=CK267Q7KZNDZHQOOCASBK4T22XW2FB3C2OHEXA3HVJVRADKKMVPQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.369668272 (498h56m30.240751307s) [noise=true , address=145.79.227.37:4566 , piece_id=OFJQA2IMHUIQERT4VO5IDJGOTS6NDQZQPX5SGPQ52HWLR6QK6C2Q, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.376976877 (498h56m30.248059912s) [noise=true , address=145.79.227.43:4157 , piece_id=XLBHS7BTQOUTHXYMMLCQRLGJAUSTBPXTVLMX7ZNPOYLIWDWTKAJQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.379469584 (498h56m30.250552619s) [noise=true , address=145.79.227.4:4185 , piece_id=UXP7AP26EF2S44AHCVTKILCT7QZS6PKXJN4G5NMNWE4V4KA7X6TQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.380055119 (498h56m30.251138165s) [noise=true , address=145.79.227.5:4222 , piece_id=BC3KZHAPQM2FX6ORY2DFTRFXVS5YXIQA5P5ZHI5MFTOY2MDU7WWA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.388429615 (498h56m30.25951265s) [noise=true , address=145.79.227.29:4942 , piece_id=3E3AAJLKD42KSYXLC7JWTFHGLXKYDAHHZLKZWCY2HHSCZ7MVMAUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.560310524 (498h56m30.431393559s) [noise=true , address=77.90.31.17:4773 , piece_id=U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.612021641 (498h56m30.483104676s) [noise=true , address=45.91.116.4:4937 , piece_id=BYNEKIDVTUEDJ3D63FF3NMJ2MNDR7BVK3PV37PM43YYMYVCXERUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.631168459 (498h56m30.502251494s) [noise=true , address=45.91.116.9:4262 , piece_id=XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.025679662 (498h56m30.896762707s) [noise=true , address=77.90.31.18:4808 , piece_id=DT4NSARC4O2YK2X7ZIOXTWNJDLC53WXOQ6T5L7SUJQEK7UUYFUUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.376755608 (498h56m31.247838643s) [noise=true , address=45.91.116.6:4974 , piece_id=TAMZQNHVRWTYHZCYNUMO7HNBMFFETLVQQVMGHC5SSKNHLB7ABGDA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.546990753 (498h56m31.418073788s) [noise=true , address=77.90.31.14:4663 , piece_id=NJJZPZDTPCJDKSSH3UMMQCXJ7ETMMG44WNUZPJHLXOPCWLVFCDAQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.709881213 (498h56m31.580964258s) [noise=true , address=77.90.31.16:4736 , piece_id=FUBKPFM2VTHHXX6JQZJRIA3LCS4A3663YJI4GKLK5BT7GF2KZYCA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.709902273 (498h56m31.580985308s) [seg_pos={0 0}, upload=437919]: Segment reached optimal threshold of 13 pieces. Dec 26 10:25:31.710110780 (498h56m31.581193825s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 10:25:31.710117674 (498h56m31.581200719s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710461367 (498h56m31.581544413s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 26 10:25:31.710479342 (498h56m31.581562387s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710598328 (498h56m31.581681363s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection Dec 26 10:25:31.710606485 (498h56m31.58168952s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710631842 (498h56m31.581714877s) [seg_pos={0 0}, upload=437919]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:25:31.710707096 (498h56m31.581790141s) [upload=437919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:25:31.710712426 (498h56m31.581795461s) [upload=437919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:25:31.998705452 (498h56m31.869788487s) [upload=437919]: Done uploading segments... Dec 26 10:25:31.998722373 (498h56m31.869805408s) [upload=437919]: Upload finished. err: <nil> Duration: 6.077138282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X......X.........X..................X.......XXX......X..] duration: 57.515957701s trace: https://jaeger.dev.storj.io/trace/7e25338f73046b2a FAIL (2025-12-26 10:27:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033130872 (498h55m22.904213918s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033471341 (498h55m22.904554386s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033892292 (498h55m22.904975328s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034387946 (498h55m22.905470991s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034807164 (498h55m22.905890209s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035259836 (498h55m22.906342881s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035727828 (498h55m22.906810863s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036033871 (498h55m22.907116906s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036579810 (498h55m22.907662855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.037767882 (498h55m22.908850927s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.038064076 (498h55m22.909147122s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.039224646 (498h55m22.910307691s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040040249 (498h55m22.911123294s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040366420 (498h55m22.911449455s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.041569130 (498h55m22.912652175s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:24.513239450 (498h55m24.384322485s) [upload=437904]: Next returned nil segment Dec 26 10:24:24.513254640 (498h55m24.384337685s) [upload=437904]: Waiting for error group managing segments... Dec 26 10:24:24.926029601 (498h55m24.797112636s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.027458408 (498h55m24.898541443s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.094888666 (498h55m24.965971701s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.102988388 (498h55m24.974071433s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.109420056 (498h55m24.980503091s) [noise=true , address=77.90.31.15:4732 , piece_id=QDQXC2LEVGS7K5OHDX7Q3ZETJJ37XFXIS6BXXNFCUTKFEMDLJ23A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.122147140 (498h55m24.993230186s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.187517306 (498h55m25.058600351s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.195537778 (498h55m25.066620823s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.255475810 (498h55m25.126558855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.546862739 (498h55m25.417945784s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.792060605 (498h55m25.66314365s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.809945911 (498h55m25.681028946s) [noise=true , address=145.79.227.44:4221 , piece_id=IZLW4HA64OMYW23ZBPBZU45GR4QCEREPUL76QL6RWWUA3LKKFN2A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.860843739 (498h55m25.731926774s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861204966 (498h55m25.732288001s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861232289 (498h55m25.732315324s) [seg_pos={0 0}, upload=437904]: Segment reached optimal threshold of 14 pieces. Dec 26 10:24:25.861471414 (498h55m25.732554449s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861479399 (498h55m25.732562434s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861580682 (498h55m25.732663717s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection Dec 26 10:24:25.861589008 (498h55m25.732672043s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861840307 (498h55m25.732923342s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861847050 (498h55m25.732930085s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861929887 (498h55m25.733012932s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861939285 (498h55m25.73302232s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861959854 (498h55m25.733042889s) [seg_pos={0 0}, upload=437904]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:24:25.862033124 (498h55m25.733116159s) [upload=437904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:24:25.862037923 (498h55m25.733120958s) [upload=437904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:24:26.151364428 (498h55m26.022447463s) [upload=437904]: Done uploading segments... Dec 26 10:24:26.151393914 (498h55m26.022476949s) [upload=437904]: Upload finished. err: <nil> Duration: 57.515974854s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X.] duration: 5.695110839s trace: https://jaeger.dev.storj.io/trace/5acd663927bec960 FAIL (2025-12-26 10:26:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ) context deadline exceeded ======LOGS====== 0:25:28.105454108 (498h56m27.976537143s) [upload=437919]: Starting upload Dec 26 10:25:28.105529822 (498h56m27.976612867s) [upload=437919]: Done starting upload Dec 26 10:25:28.105615174 (498h56m27.97669822s) [upload=437919]: Uploading segments... Dec 26 10:25:28.105756253 (498h56m27.976839298s) [upload=437919]: Got next segment. Inline: false Dec 26 10:25:28.105787723 (498h56m27.976870768s) [upload=437919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:25:28.105792612 (498h56m27.976875677s) [upload=437919]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:25:28.347683773 (498h56m28.218766808s) [seg_pos={0 0}, upload=437919]: Begin upload segment... Dec 26 10:25:28.347735281 (498h56m28.218818326s) [seg_pos={0 0}, upload=437919]: Done begin upload segment. Dec 26 10:25:28.347768394 (498h56m28.218851429s) [noise=true , address=77.90.31.17:4773 , piece_id=U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.347775288 (498h56m28.218858323s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348182993 (498h56m28.219266038s) [noise=true , address=145.79.227.4:4185 , piece_id=UXP7AP26EF2S44AHCVTKILCT7QZS6PKXJN4G5NMNWE4V4KA7X6TQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348188744 (498h56m28.219271779s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348476423 (498h56m28.219559468s) [noise=true , address=45.91.116.4:4937 , piece_id=BYNEKIDVTUEDJ3D63FF3NMJ2MNDR7BVK3PV37PM43YYMYVCXERUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348544993 (498h56m28.219628038s) [noise=true , address=145.79.227.19:4078 , piece_id=CK267Q7KZNDZHQOOCASBK4T22XW2FB3C2OHEXA3HVJVRADKKMVPQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348775251 (498h56m28.219858296s) [noise=true , address=145.79.227.37:4566 , piece_id=OFJQA2IMHUIQERT4VO5IDJGOTS6NDQZQPX5SGPQ52HWLR6QK6C2Q, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349056907 (498h56m28.220139942s) [noise=true , address=77.90.31.18:4808 , piece_id=DT4NSARC4O2YK2X7ZIOXTWNJDLC53WXOQ6T5L7SUJQEK7UUYFUUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349346079 (498h56m28.220429124s) [noise=true , address=145.79.227.5:4222 , piece_id=BC3KZHAPQM2FX6ORY2DFTRFXVS5YXIQA5P5ZHI5MFTOY2MDU7WWA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349638154 (498h56m28.220721189s) [noise=true , address=77.90.31.16:4736 , piece_id=FUBKPFM2VTHHXX6JQZJRIA3LCS4A3663YJI4GKLK5BT7GF2KZYCA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350014451 (498h56m28.221097486s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350396899 (498h56m28.221479944s) [noise=true , address=45.91.116.9:4262 , piece_id=XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350785810 (498h56m28.221868845s) [noise=true , address=45.91.116.6:4974 , piece_id=TAMZQNHVRWTYHZCYNUMO7HNBMFFETLVQQVMGHC5SSKNHLB7ABGDA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.351265854 (498h56m28.222348889s) [noise=true , address=77.90.31.14:4663 , piece_id=NJJZPZDTPCJDKSSH3UMMQCXJ7ETMMG44WNUZPJHLXOPCWLVFCDAQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.351746580 (498h56m28.222829615s) [noise=true , address=145.79.227.29:4942 , piece_id=3E3AAJLKD42KSYXLC7JWTFHGLXKYDAHHZLKZWCY2HHSCZ7MVMAUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.353707453 (498h56m28.224790498s) [noise=true , address=145.79.227.43:4157 , piece_id=XLBHS7BTQOUTHXYMMLCQRLGJAUSTBPXTVLMX7ZNPOYLIWDWTKAJQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:30.072964459 (498h56m29.944047514s) [upload=437919]: Next returned nil segment Dec 26 10:25:30.073021316 (498h56m29.944104361s) [upload=437919]: Waiting for error group managing segments... Dec 26 10:25:30.360224859 (498h56m30.231307894s) [noise=true , address=145.79.227.19:4078 , piece_id=CK267Q7KZNDZHQOOCASBK4T22XW2FB3C2OHEXA3HVJVRADKKMVPQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.369668272 (498h56m30.240751307s) [noise=true , address=145.79.227.37:4566 , piece_id=OFJQA2IMHUIQERT4VO5IDJGOTS6NDQZQPX5SGPQ52HWLR6QK6C2Q, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.376976877 (498h56m30.248059912s) [noise=true , address=145.79.227.43:4157 , piece_id=XLBHS7BTQOUTHXYMMLCQRLGJAUSTBPXTVLMX7ZNPOYLIWDWTKAJQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.379469584 (498h56m30.250552619s) [noise=true , address=145.79.227.4:4185 , piece_id=UXP7AP26EF2S44AHCVTKILCT7QZS6PKXJN4G5NMNWE4V4KA7X6TQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.380055119 (498h56m30.251138165s) [noise=true , address=145.79.227.5:4222 , piece_id=BC3KZHAPQM2FX6ORY2DFTRFXVS5YXIQA5P5ZHI5MFTOY2MDU7WWA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.388429615 (498h56m30.25951265s) [noise=true , address=145.79.227.29:4942 , piece_id=3E3AAJLKD42KSYXLC7JWTFHGLXKYDAHHZLKZWCY2HHSCZ7MVMAUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.560310524 (498h56m30.431393559s) [noise=true , address=77.90.31.17:4773 , piece_id=U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.612021641 (498h56m30.483104676s) [noise=true , address=45.91.116.4:4937 , piece_id=BYNEKIDVTUEDJ3D63FF3NMJ2MNDR7BVK3PV37PM43YYMYVCXERUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.631168459 (498h56m30.502251494s) [noise=true , address=45.91.116.9:4262 , piece_id=XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.025679662 (498h56m30.896762707s) [noise=true , address=77.90.31.18:4808 , piece_id=DT4NSARC4O2YK2X7ZIOXTWNJDLC53WXOQ6T5L7SUJQEK7UUYFUUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.376755608 (498h56m31.247838643s) [noise=true , address=45.91.116.6:4974 , piece_id=TAMZQNHVRWTYHZCYNUMO7HNBMFFETLVQQVMGHC5SSKNHLB7ABGDA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.546990753 (498h56m31.418073788s) [noise=true , address=77.90.31.14:4663 , piece_id=NJJZPZDTPCJDKSSH3UMMQCXJ7ETMMG44WNUZPJHLXOPCWLVFCDAQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.709881213 (498h56m31.580964258s) [noise=true , address=77.90.31.16:4736 , piece_id=FUBKPFM2VTHHXX6JQZJRIA3LCS4A3663YJI4GKLK5BT7GF2KZYCA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.709902273 (498h56m31.580985308s) [seg_pos={0 0}, upload=437919]: Segment reached optimal threshold of 13 pieces. Dec 26 10:25:31.710110780 (498h56m31.581193825s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 10:25:31.710117674 (498h56m31.581200719s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710461367 (498h56m31.581544413s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 26 10:25:31.710479342 (498h56m31.581562387s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710598328 (498h56m31.581681363s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection Dec 26 10:25:31.710606485 (498h56m31.58168952s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710631842 (498h56m31.581714877s) [seg_pos={0 0}, upload=437919]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:25:31.710707096 (498h56m31.581790141s) [upload=437919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:25:31.710712426 (498h56m31.581795461s) [upload=437919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:25:31.998705452 (498h56m31.869788487s) [upload=437919]: Done uploading segments... Dec 26 10:25:31.998722373 (498h56m31.869805408s) [upload=437919]: Upload finished. err: <nil> Duration: 5.695136569s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X......X.........X..................X.......XXX......X.] duration: 50.55414311s trace: https://jaeger.dev.storj.io/trace/7e25338f73046b2a FAIL (2025-12-26 10:26:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033130872 (498h55m22.904213918s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033471341 (498h55m22.904554386s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033892292 (498h55m22.904975328s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034387946 (498h55m22.905470991s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034807164 (498h55m22.905890209s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035259836 (498h55m22.906342881s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035727828 (498h55m22.906810863s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036033871 (498h55m22.907116906s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036579810 (498h55m22.907662855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.037767882 (498h55m22.908850927s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.038064076 (498h55m22.909147122s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.039224646 (498h55m22.910307691s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040040249 (498h55m22.911123294s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040366420 (498h55m22.911449455s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.041569130 (498h55m22.912652175s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:24.513239450 (498h55m24.384322485s) [upload=437904]: Next returned nil segment Dec 26 10:24:24.513254640 (498h55m24.384337685s) [upload=437904]: Waiting for error group managing segments... Dec 26 10:24:24.926029601 (498h55m24.797112636s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.027458408 (498h55m24.898541443s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.094888666 (498h55m24.965971701s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.102988388 (498h55m24.974071433s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.109420056 (498h55m24.980503091s) [noise=true , address=77.90.31.15:4732 , piece_id=QDQXC2LEVGS7K5OHDX7Q3ZETJJ37XFXIS6BXXNFCUTKFEMDLJ23A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.122147140 (498h55m24.993230186s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.187517306 (498h55m25.058600351s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.195537778 (498h55m25.066620823s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.255475810 (498h55m25.126558855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.546862739 (498h55m25.417945784s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.792060605 (498h55m25.66314365s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.809945911 (498h55m25.681028946s) [noise=true , address=145.79.227.44:4221 , piece_id=IZLW4HA64OMYW23ZBPBZU45GR4QCEREPUL76QL6RWWUA3LKKFN2A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.860843739 (498h55m25.731926774s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861204966 (498h55m25.732288001s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861232289 (498h55m25.732315324s) [seg_pos={0 0}, upload=437904]: Segment reached optimal threshold of 14 pieces. Dec 26 10:24:25.861471414 (498h55m25.732554449s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861479399 (498h55m25.732562434s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861580682 (498h55m25.732663717s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection Dec 26 10:24:25.861589008 (498h55m25.732672043s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861840307 (498h55m25.732923342s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861847050 (498h55m25.732930085s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861929887 (498h55m25.733012932s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861939285 (498h55m25.73302232s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861959854 (498h55m25.733042889s) [seg_pos={0 0}, upload=437904]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:24:25.862033124 (498h55m25.733116159s) [upload=437904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:24:25.862037923 (498h55m25.733120958s) [upload=437904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:24:26.151364428 (498h55m26.022447463s) [upload=437904]: Done uploading segments... Dec 26 10:24:26.151393914 (498h55m26.022476949s) [upload=437904]: Upload finished. err: <nil> Duration: 50.554161896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X] duration: 30.002844661s trace: https://jaeger.dev.storj.io/trace/5acd663927bec960 FAIL (2025-12-26 10:25:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ) context deadline exceeded ======LOGS====== 0:25:28.105454108 (498h56m27.976537143s) [upload=437919]: Starting upload Dec 26 10:25:28.105529822 (498h56m27.976612867s) [upload=437919]: Done starting upload Dec 26 10:25:28.105615174 (498h56m27.97669822s) [upload=437919]: Uploading segments... Dec 26 10:25:28.105756253 (498h56m27.976839298s) [upload=437919]: Got next segment. Inline: false Dec 26 10:25:28.105787723 (498h56m27.976870768s) [upload=437919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:25:28.105792612 (498h56m27.976875677s) [upload=437919]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:25:28.347683773 (498h56m28.218766808s) [seg_pos={0 0}, upload=437919]: Begin upload segment... Dec 26 10:25:28.347735281 (498h56m28.218818326s) [seg_pos={0 0}, upload=437919]: Done begin upload segment. Dec 26 10:25:28.347768394 (498h56m28.218851429s) [noise=true , address=77.90.31.17:4773 , piece_id=U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.347775288 (498h56m28.218858323s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348182993 (498h56m28.219266038s) [noise=true , address=145.79.227.4:4185 , piece_id=UXP7AP26EF2S44AHCVTKILCT7QZS6PKXJN4G5NMNWE4V4KA7X6TQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348188744 (498h56m28.219271779s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348476423 (498h56m28.219559468s) [noise=true , address=45.91.116.4:4937 , piece_id=BYNEKIDVTUEDJ3D63FF3NMJ2MNDR7BVK3PV37PM43YYMYVCXERUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348544993 (498h56m28.219628038s) [noise=true , address=145.79.227.19:4078 , piece_id=CK267Q7KZNDZHQOOCASBK4T22XW2FB3C2OHEXA3HVJVRADKKMVPQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.348775251 (498h56m28.219858296s) [noise=true , address=145.79.227.37:4566 , piece_id=OFJQA2IMHUIQERT4VO5IDJGOTS6NDQZQPX5SGPQ52HWLR6QK6C2Q, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349056907 (498h56m28.220139942s) [noise=true , address=77.90.31.18:4808 , piece_id=DT4NSARC4O2YK2X7ZIOXTWNJDLC53WXOQ6T5L7SUJQEK7UUYFUUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349346079 (498h56m28.220429124s) [noise=true , address=145.79.227.5:4222 , piece_id=BC3KZHAPQM2FX6ORY2DFTRFXVS5YXIQA5P5ZHI5MFTOY2MDU7WWA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.349638154 (498h56m28.220721189s) [noise=true , address=77.90.31.16:4736 , piece_id=FUBKPFM2VTHHXX6JQZJRIA3LCS4A3663YJI4GKLK5BT7GF2KZYCA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350014451 (498h56m28.221097486s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350396899 (498h56m28.221479944s) [noise=true , address=45.91.116.9:4262 , piece_id=XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.350785810 (498h56m28.221868845s) [noise=true , address=45.91.116.6:4974 , piece_id=TAMZQNHVRWTYHZCYNUMO7HNBMFFETLVQQVMGHC5SSKNHLB7ABGDA, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.351265854 (498h56m28.222348889s) [noise=true , address=77.90.31.14:4663 , piece_id=NJJZPZDTPCJDKSSH3UMMQCXJ7ETMMG44WNUZPJHLXOPCWLVFCDAQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.351746580 (498h56m28.222829615s) [noise=true , address=145.79.227.29:4942 , piece_id=3E3AAJLKD42KSYXLC7JWTFHGLXKYDAHHZLKZWCY2HHSCZ7MVMAUQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:28.353707453 (498h56m28.224790498s) [noise=true , address=145.79.227.43:4157 , piece_id=XLBHS7BTQOUTHXYMMLCQRLGJAUSTBPXTVLMX7ZNPOYLIWDWTKAJQ, seg_pos={0 0}, upload=437919]: Uploading piece... Dec 26 10:25:30.072964459 (498h56m29.944047514s) [upload=437919]: Next returned nil segment Dec 26 10:25:30.073021316 (498h56m29.944104361s) [upload=437919]: Waiting for error group managing segments... Dec 26 10:25:30.360224859 (498h56m30.231307894s) [noise=true , address=145.79.227.19:4078 , piece_id=CK267Q7KZNDZHQOOCASBK4T22XW2FB3C2OHEXA3HVJVRADKKMVPQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.369668272 (498h56m30.240751307s) [noise=true , address=145.79.227.37:4566 , piece_id=OFJQA2IMHUIQERT4VO5IDJGOTS6NDQZQPX5SGPQ52HWLR6QK6C2Q, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.376976877 (498h56m30.248059912s) [noise=true , address=145.79.227.43:4157 , piece_id=XLBHS7BTQOUTHXYMMLCQRLGJAUSTBPXTVLMX7ZNPOYLIWDWTKAJQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.379469584 (498h56m30.250552619s) [noise=true , address=145.79.227.4:4185 , piece_id=UXP7AP26EF2S44AHCVTKILCT7QZS6PKXJN4G5NMNWE4V4KA7X6TQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.380055119 (498h56m30.251138165s) [noise=true , address=145.79.227.5:4222 , piece_id=BC3KZHAPQM2FX6ORY2DFTRFXVS5YXIQA5P5ZHI5MFTOY2MDU7WWA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.388429615 (498h56m30.25951265s) [noise=true , address=145.79.227.29:4942 , piece_id=3E3AAJLKD42KSYXLC7JWTFHGLXKYDAHHZLKZWCY2HHSCZ7MVMAUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.560310524 (498h56m30.431393559s) [noise=true , address=77.90.31.17:4773 , piece_id=U3B6DODAZC4SWIG4MTDPNSLBH4G6ZHR2MARLCPGWSG3KLH23MDHQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.612021641 (498h56m30.483104676s) [noise=true , address=45.91.116.4:4937 , piece_id=BYNEKIDVTUEDJ3D63FF3NMJ2MNDR7BVK3PV37PM43YYMYVCXERUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:30.631168459 (498h56m30.502251494s) [noise=true , address=45.91.116.9:4262 , piece_id=XB4X724QLIOHHHXB4IMTIIXTRMZKKOO5H4EKLUX5WMCJ4Q73DZ2Q, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.025679662 (498h56m30.896762707s) [noise=true , address=77.90.31.18:4808 , piece_id=DT4NSARC4O2YK2X7ZIOXTWNJDLC53WXOQ6T5L7SUJQEK7UUYFUUQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.376755608 (498h56m31.247838643s) [noise=true , address=45.91.116.6:4974 , piece_id=TAMZQNHVRWTYHZCYNUMO7HNBMFFETLVQQVMGHC5SSKNHLB7ABGDA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.546990753 (498h56m31.418073788s) [noise=true , address=77.90.31.14:4663 , piece_id=NJJZPZDTPCJDKSSH3UMMQCXJ7ETMMG44WNUZPJHLXOPCWLVFCDAQ, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.709881213 (498h56m31.580964258s) [noise=true , address=77.90.31.16:4736 , piece_id=FUBKPFM2VTHHXX6JQZJRIA3LCS4A3663YJI4GKLK5BT7GF2KZYCA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: <nil> Dec 26 10:25:31.709902273 (498h56m31.580985308s) [seg_pos={0 0}, upload=437919]: Segment reached optimal threshold of 13 pieces. Dec 26 10:25:31.710110780 (498h56m31.581193825s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 10:25:31.710117674 (498h56m31.581200719s) [noise=true , address=145.79.227.44:4194 , piece_id=YPH7GYSDPEZY5TTK4K5G3DXINDOGFHIUXV7FRORHDTERL3SG4PBA, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710461367 (498h56m31.581544413s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 26 10:25:31.710479342 (498h56m31.581562387s) [noise=true , address=77.90.31.15:4700 , piece_id=DAMPJ2CUQOOIWNFON4TEJVSMYFNGIKTYXOFDQQEE2Y5JY4H3HB5A, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710598328 (498h56m31.581681363s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59304->45.91.116.14:4191: use of closed network connection Dec 26 10:25:31.710606485 (498h56m31.58168952s) [noise=true , address=45.91.116.14:4191 , piece_id=OFBASIKIQ2YGTU2CWFDIC5C35QC7EYSJ2Z6ESUKI76K7IOJ3OOSA, seg_pos={0 0}, upload=437919]: Long tail cancellation due to optimal threshold Dec 26 10:25:31.710631842 (498h56m31.581714877s) [seg_pos={0 0}, upload=437919]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:25:31.710707096 (498h56m31.581790141s) [upload=437919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:25:31.710712426 (498h56m31.581795461s) [upload=437919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:25:31.998705452 (498h56m31.869788487s) [upload=437919]: Done uploading segments... Dec 26 10:25:31.998722373 (498h56m31.869805408s) [upload=437919]: Upload finished. err: <nil> Duration: 30.002895368s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X......X.........X..................X.......XXX......X] duration: 1m0.002184177s trace: https://jaeger.dev.storj.io/trace/7e25338f73046b2a FAIL (2025-12-26 10:25:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033130872 (498h55m22.904213918s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033471341 (498h55m22.904554386s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033892292 (498h55m22.904975328s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034387946 (498h55m22.905470991s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034807164 (498h55m22.905890209s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035259836 (498h55m22.906342881s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035727828 (498h55m22.906810863s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036033871 (498h55m22.907116906s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036579810 (498h55m22.907662855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.037767882 (498h55m22.908850927s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.038064076 (498h55m22.909147122s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.039224646 (498h55m22.910307691s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040040249 (498h55m22.911123294s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040366420 (498h55m22.911449455s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.041569130 (498h55m22.912652175s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:24.513239450 (498h55m24.384322485s) [upload=437904]: Next returned nil segment Dec 26 10:24:24.513254640 (498h55m24.384337685s) [upload=437904]: Waiting for error group managing segments... Dec 26 10:24:24.926029601 (498h55m24.797112636s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.027458408 (498h55m24.898541443s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.094888666 (498h55m24.965971701s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.102988388 (498h55m24.974071433s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.109420056 (498h55m24.980503091s) [noise=true , address=77.90.31.15:4732 , piece_id=QDQXC2LEVGS7K5OHDX7Q3ZETJJ37XFXIS6BXXNFCUTKFEMDLJ23A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.122147140 (498h55m24.993230186s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.187517306 (498h55m25.058600351s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.195537778 (498h55m25.066620823s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.255475810 (498h55m25.126558855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.546862739 (498h55m25.417945784s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.792060605 (498h55m25.66314365s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.809945911 (498h55m25.681028946s) [noise=true , address=145.79.227.44:4221 , piece_id=IZLW4HA64OMYW23ZBPBZU45GR4QCEREPUL76QL6RWWUA3LKKFN2A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.860843739 (498h55m25.731926774s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861204966 (498h55m25.732288001s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861232289 (498h55m25.732315324s) [seg_pos={0 0}, upload=437904]: Segment reached optimal threshold of 14 pieces. Dec 26 10:24:25.861471414 (498h55m25.732554449s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861479399 (498h55m25.732562434s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861580682 (498h55m25.732663717s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection Dec 26 10:24:25.861589008 (498h55m25.732672043s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861840307 (498h55m25.732923342s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861847050 (498h55m25.732930085s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861929887 (498h55m25.733012932s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861939285 (498h55m25.73302232s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861959854 (498h55m25.733042889s) [seg_pos={0 0}, upload=437904]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:24:25.862033124 (498h55m25.733116159s) [upload=437904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:24:25.862037923 (498h55m25.733120958s) [upload=437904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:24:26.151364428 (498h55m26.022447463s) [upload=437904]: Done uploading segments... Dec 26 10:24:26.151393914 (498h55m26.022476949s) [upload=437904]: Upload finished. err: <nil> Duration: 1m0.002195267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.] duration: 6.509569803s trace: https://jaeger.dev.storj.io/trace/7eede2270b4737e FAIL (2025-12-26 10:24:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: DUFAPFMVHBS4DWTNPPYQ3J6FYUCJDBFV7M2K6JWSOTUKOOKW2KAQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: MIWG3AJUIRVEKN7VKH5IKRCSHPC2SOUXK46GMSFSBXFBOWOTATKA) context deadline exceeded ======LOGS====== :23:25.838044470 (498h54m25.709127515s) [upload=437893]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:23:26.084651240 (498h54m25.955734275s) [seg_pos={0 0}, upload=437893]: Begin upload segment... Dec 26 10:23:26.084705242 (498h54m25.955788287s) [seg_pos={0 0}, upload=437893]: Done begin upload segment. Dec 26 10:23:26.084736121 (498h54m25.955819156s) [noise=true , address=145.79.227.52:4445 , piece_id=WMXSZHOA6QE5OURIWYRVNCZTPQYPP2HCOS6PM3RRDGOC35CQ62EA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.084844357 (498h54m25.955927392s) [noise=true , address=145.79.227.19:4078 , piece_id=4FPPCEXBLJBOR3SN7LHAZM5VYIP2CUQN5EV3RPPCX5DF2PQ3HXTQ, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.085457855 (498h54m25.9565409s) [noise=true , address=145.79.227.34:4907 , piece_id=EPNMLFRJTUNSBBTOWHQJH5DEGJ2CVSNFWDJ2FQGFZV36MOAND6CA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.086013113 (498h54m25.957096148s) [noise=true , address=45.91.116.8:4047 , piece_id=BL7DQIFISVRTJIMEJBJDPDD7CH5AHUVMKBBN4TE42LQ73GUU4AQA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.086651528 (498h54m25.957734563s) [noise=true , address=45.91.116.12:4225 , piece_id=MIWG3AJUIRVEKN7VKH5IKRCSHPC2SOUXK46GMSFSBXFBOWOTATKA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.087050438 (498h54m25.958133473s) [noise=true , address=45.91.116.6:4973 , piece_id=JSA2Y2SQREHGFQ7E52KFMK7TDNVVEXGTNMNKVCQ6MAIOTOGJFDEA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.087466110 (498h54m25.958549155s) [noise=true , address=145.79.227.16:4909 , piece_id=TSXL4A2YTUJZNV25POC3NJOANRGH676O2O6JV44WCISXSNQ6XPAA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.088465783 (498h54m25.959548818s) [noise=true , address=145.79.227.24:4583 , piece_id=3JU37TO3IZZVLR5LVXTJ6HIOBP3YHF57P7P2XC6Z5DDB46F7NI7Q, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.088827231 (498h54m25.959910277s) [noise=true , address=145.79.227.47:4267 , piece_id=K3NCH2X27LKEWWPZU2S5MDNW5QRZD7ZXDKGMHTCQ4D52IA4DM2OQ, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.089631984 (498h54m25.960715019s) [noise=true , address=45.91.116.4:4939 , piece_id=AKCRBKNXKYSP2N4BLMTH3RIIKUXYHH7DGQHHWFQX332GW3HFUREA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.090325565 (498h54m25.96140861s) [noise=true , address=45.91.116.11:4118 , piece_id=LDFZMN3FY6CAELQD4GRXZLQNXSNZQUKIZEKXWLACOWICYJSRHGUQ, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.090925958 (498h54m25.962008993s) [noise=true , address=77.90.31.16:4736 , piece_id=JMKQ2BHRCM6C3GCR65GAROVQIR66LDSJ5LYF6FGJJY4KCIRT64EA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.092056019 (498h54m25.963139054s) [noise=true , address=145.79.227.2:4115 , piece_id=BMGC5S7GSYFQQKAJBIXQSBRX3B5WSVXOJGNQIWF2Y3IRAGNF4NUQ, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.093164209 (498h54m25.964247244s) [noise=true , address=45.91.116.13:4156 , piece_id=DUFAPFMVHBS4DWTNPPYQ3J6FYUCJDBFV7M2K6JWSOTUKOOKW2KAQ, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.093417522 (498h54m25.964500557s) [noise=true , address=145.79.227.42:4122 , piece_id=4YTJTN3ROQD4QCAFEIWU7IDGUW6SINQESD3DOIXJULZFEJMSHDJQ, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.095203363 (498h54m25.966286408s) [noise=true , address=45.91.116.9:4261 , piece_id=M6NXM66YG6BFTKHB4YIAXGWYCUNQUFLBIQCDSZNFHHDWH6EW45RQ, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:27.803273165 (498h54m27.6743562s) [upload=437893]: Next returned nil segment Dec 26 10:23:27.803285979 (498h54m27.674369014s) [upload=437893]: Waiting for error group managing segments... Dec 26 10:23:28.090556359 (498h54m27.961639394s) [noise=true , address=145.79.227.19:4078 , piece_id=4FPPCEXBLJBOR3SN7LHAZM5VYIP2CUQN5EV3RPPCX5DF2PQ3HXTQ, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.093463165 (498h54m27.9645462s) [noise=true , address=145.79.227.2:4115 , piece_id=BMGC5S7GSYFQQKAJBIXQSBRX3B5WSVXOJGNQIWF2Y3IRAGNF4NUQ, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.094062746 (498h54m27.965145781s) [noise=true , address=145.79.227.52:4445 , piece_id=WMXSZHOA6QE5OURIWYRVNCZTPQYPP2HCOS6PM3RRDGOC35CQ62EA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.096537338 (498h54m27.967620383s) [noise=true , address=145.79.227.34:4907 , piece_id=EPNMLFRJTUNSBBTOWHQJH5DEGJ2CVSNFWDJ2FQGFZV36MOAND6CA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.105127555 (498h54m27.97621059s) [noise=true , address=145.79.227.42:4122 , piece_id=4YTJTN3ROQD4QCAFEIWU7IDGUW6SINQESD3DOIXJULZFEJMSHDJQ, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.111612093 (498h54m27.982695128s) [noise=true , address=145.79.227.47:4267 , piece_id=K3NCH2X27LKEWWPZU2S5MDNW5QRZD7ZXDKGMHTCQ4D52IA4DM2OQ, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.116766007 (498h54m27.987849052s) [noise=true , address=145.79.227.24:4583 , piece_id=3JU37TO3IZZVLR5LVXTJ6HIOBP3YHF57P7P2XC6Z5DDB46F7NI7Q, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.117941686 (498h54m27.989024721s) [noise=true , address=145.79.227.16:4909 , piece_id=TSXL4A2YTUJZNV25POC3NJOANRGH676O2O6JV44WCISXSNQ6XPAA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.332607534 (498h54m28.203690619s) [noise=true , address=45.91.116.13:4156 , piece_id=DUFAPFMVHBS4DWTNPPYQ3J6FYUCJDBFV7M2K6JWSOTUKOOKW2KAQ, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.479856660 (498h54m28.350939705s) [noise=true , address=77.90.31.16:4736 , piece_id=JMKQ2BHRCM6C3GCR65GAROVQIR66LDSJ5LYF6FGJJY4KCIRT64EA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.545923221 (498h54m28.417006256s) [noise=true , address=45.91.116.11:4118 , piece_id=LDFZMN3FY6CAELQD4GRXZLQNXSNZQUKIZEKXWLACOWICYJSRHGUQ, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.667211123 (498h54m28.538294168s) [noise=true , address=45.91.116.4:4939 , piece_id=AKCRBKNXKYSP2N4BLMTH3RIIKUXYHH7DGQHHWFQX332GW3HFUREA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.683526130 (498h54m28.554609165s) [noise=true , address=45.91.116.12:4225 , piece_id=MIWG3AJUIRVEKN7VKH5IKRCSHPC2SOUXK46GMSFSBXFBOWOTATKA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.683547410 (498h54m28.554630445s) [seg_pos={0 0}, upload=437893]: Segment reached optimal threshold of 13 pieces. Dec 26 10:23:28.683719908 (498h54m28.554802943s) [noise=true , address=45.91.116.9:4261 , piece_id=M6NXM66YG6BFTKHB4YIAXGWYCUNQUFLBIQCDSZNFHHDWH6EW45RQ, seg_pos={0 0}, upload=437893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:56416->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:56416->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56416->45.91.116.9:4261: use of closed network connection Dec 26 10:23:28.683726491 (498h54m28.554809536s) [noise=true , address=45.91.116.9:4261 , piece_id=M6NXM66YG6BFTKHB4YIAXGWYCUNQUFLBIQCDSZNFHHDWH6EW45RQ, seg_pos={0 0}, upload=437893]: Long tail cancellation due to optimal threshold Dec 26 10:23:28.683940659 (498h54m28.555023694s) [noise=true , address=45.91.116.8:4047 , piece_id=BL7DQIFISVRTJIMEJBJDPDD7CH5AHUVMKBBN4TE42LQ73GUU4AQA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:37340->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:37340->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37340->45.91.116.8:4047: use of closed network connection Dec 26 10:23:28.683947412 (498h54m28.555030447s) [noise=true , address=45.91.116.8:4047 , piece_id=BL7DQIFISVRTJIMEJBJDPDD7CH5AHUVMKBBN4TE42LQ73GUU4AQA, seg_pos={0 0}, upload=437893]: Long tail cancellation due to optimal threshold Dec 26 10:23:28.684021413 (498h54m28.555104448s) [noise=true , address=45.91.116.6:4973 , piece_id=JSA2Y2SQREHGFQ7E52KFMK7TDNVVEXGTNMNKVCQ6MAIOTOGJFDEA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:55608->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:55608->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55608->45.91.116.6:4973: use of closed network connection Dec 26 10:23:28.684026112 (498h54m28.555109147s) [noise=true , address=45.91.116.6:4973 , piece_id=JSA2Y2SQREHGFQ7E52KFMK7TDNVVEXGTNMNKVCQ6MAIOTOGJFDEA, seg_pos={0 0}, upload=437893]: Long tail cancellation due to optimal threshold Dec 26 10:23:28.684044257 (498h54m28.555127302s) [seg_pos={0 0}, upload=437893]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:23:28.684107757 (498h54m28.555190792s) [upload=437893]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:23:28.684113008 (498h54m28.555196073s) [upload=437893]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:23:28.971910780 (498h54m28.842993815s) [upload=437893]: Done uploading segments... Dec 26 10:23:28.971929205 (498h54m28.84301224s) [upload=437893]: Upload finished. err: <nil> Duration:* 6.50959462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X......X.........X..................X.......XXX......] duration: 32.656422672s trace: https://jaeger.dev.storj.io/trace/7aef545d37216976 FAIL (2025-12-26 10:24:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:4883 , piece_id=45ESRA6YTW43MKEFIE2U3HPNW7JTPXM2SLNTHKY32F2YFORNVAQQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.441584147 (498h49m27.312667192s) [noise=true , address=64.49.13.137:4503 , piece_id=ZVP3DFOTW5GIW6WZ4KDQVV6GQV5MUS5VWLU5FDCAGFYFDDT2D5RA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.441837960 (498h49m27.312921005s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442146617 (498h49m27.313229653s) [noise=true , address=77.90.31.11:4560 , piece_id=G4EXQLB4I3GFTTLZNRTPTWRY2HUIBAMOTAAM333OJQB2SNZLQW7Q, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442525168 (498h49m27.313608214s) [noise=true , address=145.79.227.52:4458 , piece_id=COMKCWPTVIBRPOPEWIDYU73EFQFXWVQY7IUFEM42HRQWXGJOUZ7A, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442608046 (498h49m27.313691091s) [noise=true , address=77.90.31.15:4730 , piece_id=RMDDZY2KUS6A4SLQW2B6JTHANDBCHGRWJIRHQQJ4X635I53B3ZHQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442903218 (498h49m27.313986264s) [noise=true , address=145.79.227.34:4915 , piece_id=ADW24WPJ5YJ73F5ABFTK3QSON3TBWUMZNCWNNBMV6ODFWNRFGX5A, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442912065 (498h49m27.3139951s) [noise=true , address=77.90.31.18:4837 , piece_id=OC6GNBCZQMJCV7JSF7XHO3YHE7HMZ3HBWBKBYJRDKLMKMCYN2UFQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443184915 (498h49m27.31426796s) [noise=true , address=77.90.31.14:4680 , piece_id=JK2NZRA5QXJJ2KWIQ3KPZT2GK2RJ3R6YGWVBLUHLGDYKBWUQKCIQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443385607 (498h49m27.314468652s) [noise=true , address=145.79.227.33:4876 , piece_id=OCV2FU5GXO5Q5TZ7CQ4WPGHSZ2MWFAFCP3VEX7TVGHELVNZJIGFA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443486720 (498h49m27.314569765s) [noise=true , address=45.91.116.12:4245 , piece_id=B5SSUXAQ7FF2WZKQGROIYQSZAMYR4M7K5WWA64O7KY46TEWWZQPQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443777884 (498h49m27.314860919s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444156695 (498h49m27.31523973s) [noise=true , address=77.90.31.16:4762 , piece_id=TO4OWX4NFCLFXXPW7NQWJ7NVXWG24TTBXN4ISBZE667U27FPTXQA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444525026 (498h49m27.315608101s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444959794 (498h49m27.31604284s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:28.898009100 (498h49m28.769092155s) [upload=437819]: Next returned nil segment Dec 26 10:18:28.898024930 (498h49m28.769107976s) [upload=437819]: Waiting for error group managing segments... Dec 26 10:18:29.317608319 (498h49m29.188691364s) [noise=true , address=145.79.227.52:4458 , piece_id=COMKCWPTVIBRPOPEWIDYU73EFQFXWVQY7IUFEM42HRQWXGJOUZ7A, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.417048480 (498h49m29.288131515s) [noise=true , address=64.49.13.137:4503 , piece_id=ZVP3DFOTW5GIW6WZ4KDQVV6GQV5MUS5VWLU5FDCAGFYFDDT2D5RA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.431472206 (498h49m29.302555241s) [noise=true , address=145.79.227.34:4915 , piece_id=ADW24WPJ5YJ73F5ABFTK3QSON3TBWUMZNCWNNBMV6ODFWNRFGX5A, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.531132256 (498h49m29.402215291s) [noise=true , address=77.90.31.14:4680 , piece_id=JK2NZRA5QXJJ2KWIQ3KPZT2GK2RJ3R6YGWVBLUHLGDYKBWUQKCIQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.539874212 (498h49m29.410957247s) [noise=true , address=45.91.116.12:4245 , piece_id=B5SSUXAQ7FF2WZKQGROIYQSZAMYR4M7K5WWA64O7KY46TEWWZQPQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.575613335 (498h49m29.44669638s) [noise=true , address=77.90.31.21:4883 , piece_id=45ESRA6YTW43MKEFIE2U3HPNW7JTPXM2SLNTHKY32F2YFORNVAQQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.588530461 (498h49m29.459613496s) [noise=true , address=77.90.31.17:4780 , piece_id=OXLKBKFLRXJFLRHIQ3ROAYBDCJUBZRIZ6U3RCZJOGK7Q7BXG26CQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.606622181 (498h49m29.477705216s) [noise=true , address=145.79.227.33:4876 , piece_id=OCV2FU5GXO5Q5TZ7CQ4WPGHSZ2MWFAFCP3VEX7TVGHELVNZJIGFA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.618087524 (498h49m29.489170559s) [noise=true , address=77.90.31.16:4762 , piece_id=TO4OWX4NFCLFXXPW7NQWJ7NVXWG24TTBXN4ISBZE667U27FPTXQA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.618804549 (498h49m29.489887594s) [noise=true , address=77.90.31.15:4730 , piece_id=RMDDZY2KUS6A4SLQW2B6JTHANDBCHGRWJIRHQQJ4X635I53B3ZHQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.627317289 (498h49m29.498400324s) [noise=true , address=77.90.31.18:4837 , piece_id=OC6GNBCZQMJCV7JSF7XHO3YHE7HMZ3HBWBKBYJRDKLMKMCYN2UFQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.743470775 (498h49m29.61455381s) [noise=true , address=77.90.31.19:4863 , piece_id=OQ7UPZNHXMG52A52UUIMOBDRI4B2EQE5VURHZM4PCOKYHVAG2C2Q, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.988678213 (498h49m29.859761248s) [noise=true , address=77.90.31.11:4560 , piece_id=G4EXQLB4I3GFTTLZNRTPTWRY2HUIBAMOTAAM333OJQB2SNZLQW7Q, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:30.173673986 (498h49m30.044757031s) [noise=true , address=145.79.227.44:4222 , piece_id=XU4TQPQXW4CXIJ3SI634Z32MMJB7GFQFHKNBZEGNHVONUQSVTSYQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:30.173691770 (498h49m30.044774815s) [seg_pos={0 0}, upload=437819]: Segment reached optimal threshold of 14 pieces. Dec 26 10:18:30.173872264 (498h49m30.044955299s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:30.173880039 (498h49m30.044963084s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174008883 (498h49m30.045091918s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection Dec 26 10:18:30.174017961 (498h49m30.045100996s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174049742 (498h49m30.045132777s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:30.174058829 (498h49m30.045141864s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174254883 (498h49m30.045337918s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jEyiUUU9NCVFMU8frNggxTTYBMkehsqN4GLsFRkrxubdobGY2): protocol: write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection; write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection Dec 26 10:18:30.174261886 (498h49m30.045344921s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174347448 (498h49m30.045430493s) [seg_pos={0 0}, upload=437819]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:18:30.174415378 (498h49m30.045498413s) [upload=437819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:30.174419666 (498h49m30.045502701s) [upload=437819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:30.472374192 (498h49m30.343457227s) [upload=437819]: Done uploading segments... Dec 26 10:18:30.472393109 (498h49m30.343476144s) [upload=437819]: Upload finished. err: <nil> Duration: 32.656436448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X] duration: 30.005097019s trace: https://jaeger.dev.storj.io/trace/7eede2270b4737e FAIL (2025-12-26 10:23:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: DUFAPFMVHBS4DWTNPPYQ3J6FYUCJDBFV7M2K6JWSOTUKOOKW2KAQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: MIWG3AJUIRVEKN7VKH5IKRCSHPC2SOUXK46GMSFSBXFBOWOTATKA) context deadline exceeded ======LOGS====== :23:25.838044470 (498h54m25.709127515s) [upload=437893]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:23:26.084651240 (498h54m25.955734275s) [seg_pos={0 0}, upload=437893]: Begin upload segment... Dec 26 10:23:26.084705242 (498h54m25.955788287s) [seg_pos={0 0}, upload=437893]: Done begin upload segment. Dec 26 10:23:26.084736121 (498h54m25.955819156s) [noise=true , address=145.79.227.52:4445 , piece_id=WMXSZHOA6QE5OURIWYRVNCZTPQYPP2HCOS6PM3RRDGOC35CQ62EA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.084844357 (498h54m25.955927392s) [noise=true , address=145.79.227.19:4078 , piece_id=4FPPCEXBLJBOR3SN7LHAZM5VYIP2CUQN5EV3RPPCX5DF2PQ3HXTQ, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.085457855 (498h54m25.9565409s) [noise=true , address=145.79.227.34:4907 , piece_id=EPNMLFRJTUNSBBTOWHQJH5DEGJ2CVSNFWDJ2FQGFZV36MOAND6CA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.086013113 (498h54m25.957096148s) [noise=true , address=45.91.116.8:4047 , piece_id=BL7DQIFISVRTJIMEJBJDPDD7CH5AHUVMKBBN4TE42LQ73GUU4AQA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.086651528 (498h54m25.957734563s) [noise=true , address=45.91.116.12:4225 , piece_id=MIWG3AJUIRVEKN7VKH5IKRCSHPC2SOUXK46GMSFSBXFBOWOTATKA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.087050438 (498h54m25.958133473s) [noise=true , address=45.91.116.6:4973 , piece_id=JSA2Y2SQREHGFQ7E52KFMK7TDNVVEXGTNMNKVCQ6MAIOTOGJFDEA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.087466110 (498h54m25.958549155s) [noise=true , address=145.79.227.16:4909 , piece_id=TSXL4A2YTUJZNV25POC3NJOANRGH676O2O6JV44WCISXSNQ6XPAA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.088465783 (498h54m25.959548818s) [noise=true , address=145.79.227.24:4583 , piece_id=3JU37TO3IZZVLR5LVXTJ6HIOBP3YHF57P7P2XC6Z5DDB46F7NI7Q, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.088827231 (498h54m25.959910277s) [noise=true , address=145.79.227.47:4267 , piece_id=K3NCH2X27LKEWWPZU2S5MDNW5QRZD7ZXDKGMHTCQ4D52IA4DM2OQ, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.089631984 (498h54m25.960715019s) [noise=true , address=45.91.116.4:4939 , piece_id=AKCRBKNXKYSP2N4BLMTH3RIIKUXYHH7DGQHHWFQX332GW3HFUREA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.090325565 (498h54m25.96140861s) [noise=true , address=45.91.116.11:4118 , piece_id=LDFZMN3FY6CAELQD4GRXZLQNXSNZQUKIZEKXWLACOWICYJSRHGUQ, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.090925958 (498h54m25.962008993s) [noise=true , address=77.90.31.16:4736 , piece_id=JMKQ2BHRCM6C3GCR65GAROVQIR66LDSJ5LYF6FGJJY4KCIRT64EA, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.092056019 (498h54m25.963139054s) [noise=true , address=145.79.227.2:4115 , piece_id=BMGC5S7GSYFQQKAJBIXQSBRX3B5WSVXOJGNQIWF2Y3IRAGNF4NUQ, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.093164209 (498h54m25.964247244s) [noise=true , address=45.91.116.13:4156 , piece_id=DUFAPFMVHBS4DWTNPPYQ3J6FYUCJDBFV7M2K6JWSOTUKOOKW2KAQ, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.093417522 (498h54m25.964500557s) [noise=true , address=145.79.227.42:4122 , piece_id=4YTJTN3ROQD4QCAFEIWU7IDGUW6SINQESD3DOIXJULZFEJMSHDJQ, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:26.095203363 (498h54m25.966286408s) [noise=true , address=45.91.116.9:4261 , piece_id=M6NXM66YG6BFTKHB4YIAXGWYCUNQUFLBIQCDSZNFHHDWH6EW45RQ, seg_pos={0 0}, upload=437893]: Uploading piece... Dec 26 10:23:27.803273165 (498h54m27.6743562s) [upload=437893]: Next returned nil segment Dec 26 10:23:27.803285979 (498h54m27.674369014s) [upload=437893]: Waiting for error group managing segments... Dec 26 10:23:28.090556359 (498h54m27.961639394s) [noise=true , address=145.79.227.19:4078 , piece_id=4FPPCEXBLJBOR3SN7LHAZM5VYIP2CUQN5EV3RPPCX5DF2PQ3HXTQ, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.093463165 (498h54m27.9645462s) [noise=true , address=145.79.227.2:4115 , piece_id=BMGC5S7GSYFQQKAJBIXQSBRX3B5WSVXOJGNQIWF2Y3IRAGNF4NUQ, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.094062746 (498h54m27.965145781s) [noise=true , address=145.79.227.52:4445 , piece_id=WMXSZHOA6QE5OURIWYRVNCZTPQYPP2HCOS6PM3RRDGOC35CQ62EA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.096537338 (498h54m27.967620383s) [noise=true , address=145.79.227.34:4907 , piece_id=EPNMLFRJTUNSBBTOWHQJH5DEGJ2CVSNFWDJ2FQGFZV36MOAND6CA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.105127555 (498h54m27.97621059s) [noise=true , address=145.79.227.42:4122 , piece_id=4YTJTN3ROQD4QCAFEIWU7IDGUW6SINQESD3DOIXJULZFEJMSHDJQ, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.111612093 (498h54m27.982695128s) [noise=true , address=145.79.227.47:4267 , piece_id=K3NCH2X27LKEWWPZU2S5MDNW5QRZD7ZXDKGMHTCQ4D52IA4DM2OQ, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.116766007 (498h54m27.987849052s) [noise=true , address=145.79.227.24:4583 , piece_id=3JU37TO3IZZVLR5LVXTJ6HIOBP3YHF57P7P2XC6Z5DDB46F7NI7Q, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.117941686 (498h54m27.989024721s) [noise=true , address=145.79.227.16:4909 , piece_id=TSXL4A2YTUJZNV25POC3NJOANRGH676O2O6JV44WCISXSNQ6XPAA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.332607534 (498h54m28.203690619s) [noise=true , address=45.91.116.13:4156 , piece_id=DUFAPFMVHBS4DWTNPPYQ3J6FYUCJDBFV7M2K6JWSOTUKOOKW2KAQ, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.479856660 (498h54m28.350939705s) [noise=true , address=77.90.31.16:4736 , piece_id=JMKQ2BHRCM6C3GCR65GAROVQIR66LDSJ5LYF6FGJJY4KCIRT64EA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.545923221 (498h54m28.417006256s) [noise=true , address=45.91.116.11:4118 , piece_id=LDFZMN3FY6CAELQD4GRXZLQNXSNZQUKIZEKXWLACOWICYJSRHGUQ, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.667211123 (498h54m28.538294168s) [noise=true , address=45.91.116.4:4939 , piece_id=AKCRBKNXKYSP2N4BLMTH3RIIKUXYHH7DGQHHWFQX332GW3HFUREA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.683526130 (498h54m28.554609165s) [noise=true , address=45.91.116.12:4225 , piece_id=MIWG3AJUIRVEKN7VKH5IKRCSHPC2SOUXK46GMSFSBXFBOWOTATKA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: <nil> Dec 26 10:23:28.683547410 (498h54m28.554630445s) [seg_pos={0 0}, upload=437893]: Segment reached optimal threshold of 13 pieces. Dec 26 10:23:28.683719908 (498h54m28.554802943s) [noise=true , address=45.91.116.9:4261 , piece_id=M6NXM66YG6BFTKHB4YIAXGWYCUNQUFLBIQCDSZNFHHDWH6EW45RQ, seg_pos={0 0}, upload=437893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:56416->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:56416->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56416->45.91.116.9:4261: use of closed network connection Dec 26 10:23:28.683726491 (498h54m28.554809536s) [noise=true , address=45.91.116.9:4261 , piece_id=M6NXM66YG6BFTKHB4YIAXGWYCUNQUFLBIQCDSZNFHHDWH6EW45RQ, seg_pos={0 0}, upload=437893]: Long tail cancellation due to optimal threshold Dec 26 10:23:28.683940659 (498h54m28.555023694s) [noise=true , address=45.91.116.8:4047 , piece_id=BL7DQIFISVRTJIMEJBJDPDD7CH5AHUVMKBBN4TE42LQ73GUU4AQA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:37340->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:37340->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37340->45.91.116.8:4047: use of closed network connection Dec 26 10:23:28.683947412 (498h54m28.555030447s) [noise=true , address=45.91.116.8:4047 , piece_id=BL7DQIFISVRTJIMEJBJDPDD7CH5AHUVMKBBN4TE42LQ73GUU4AQA, seg_pos={0 0}, upload=437893]: Long tail cancellation due to optimal threshold Dec 26 10:23:28.684021413 (498h54m28.555104448s) [noise=true , address=45.91.116.6:4973 , piece_id=JSA2Y2SQREHGFQ7E52KFMK7TDNVVEXGTNMNKVCQ6MAIOTOGJFDEA, seg_pos={0 0}, upload=437893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:55608->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:55608->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55608->45.91.116.6:4973: use of closed network connection Dec 26 10:23:28.684026112 (498h54m28.555109147s) [noise=true , address=45.91.116.6:4973 , piece_id=JSA2Y2SQREHGFQ7E52KFMK7TDNVVEXGTNMNKVCQ6MAIOTOGJFDEA, seg_pos={0 0}, upload=437893]: Long tail cancellation due to optimal threshold Dec 26 10:23:28.684044257 (498h54m28.555127302s) [seg_pos={0 0}, upload=437893]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:23:28.684107757 (498h54m28.555190792s) [upload=437893]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:23:28.684113008 (498h54m28.555196073s) [upload=437893]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:23:28.971910780 (498h54m28.842993815s) [upload=437893]: Done uploading segments... Dec 26 10:23:28.971929205 (498h54m28.84301224s) [upload=437893]: Upload finished. err: <nil> Duration:* 30.005130583s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X......X.........X..................X.......XXX.....] duration: 49.311107459s trace: https://jaeger.dev.storj.io/trace/7aef545d37216976 FAIL (2025-12-26 10:23:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:4883 , piece_id=45ESRA6YTW43MKEFIE2U3HPNW7JTPXM2SLNTHKY32F2YFORNVAQQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.441584147 (498h49m27.312667192s) [noise=true , address=64.49.13.137:4503 , piece_id=ZVP3DFOTW5GIW6WZ4KDQVV6GQV5MUS5VWLU5FDCAGFYFDDT2D5RA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.441837960 (498h49m27.312921005s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442146617 (498h49m27.313229653s) [noise=true , address=77.90.31.11:4560 , piece_id=G4EXQLB4I3GFTTLZNRTPTWRY2HUIBAMOTAAM333OJQB2SNZLQW7Q, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442525168 (498h49m27.313608214s) [noise=true , address=145.79.227.52:4458 , piece_id=COMKCWPTVIBRPOPEWIDYU73EFQFXWVQY7IUFEM42HRQWXGJOUZ7A, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442608046 (498h49m27.313691091s) [noise=true , address=77.90.31.15:4730 , piece_id=RMDDZY2KUS6A4SLQW2B6JTHANDBCHGRWJIRHQQJ4X635I53B3ZHQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442903218 (498h49m27.313986264s) [noise=true , address=145.79.227.34:4915 , piece_id=ADW24WPJ5YJ73F5ABFTK3QSON3TBWUMZNCWNNBMV6ODFWNRFGX5A, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442912065 (498h49m27.3139951s) [noise=true , address=77.90.31.18:4837 , piece_id=OC6GNBCZQMJCV7JSF7XHO3YHE7HMZ3HBWBKBYJRDKLMKMCYN2UFQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443184915 (498h49m27.31426796s) [noise=true , address=77.90.31.14:4680 , piece_id=JK2NZRA5QXJJ2KWIQ3KPZT2GK2RJ3R6YGWVBLUHLGDYKBWUQKCIQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443385607 (498h49m27.314468652s) [noise=true , address=145.79.227.33:4876 , piece_id=OCV2FU5GXO5Q5TZ7CQ4WPGHSZ2MWFAFCP3VEX7TVGHELVNZJIGFA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443486720 (498h49m27.314569765s) [noise=true , address=45.91.116.12:4245 , piece_id=B5SSUXAQ7FF2WZKQGROIYQSZAMYR4M7K5WWA64O7KY46TEWWZQPQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443777884 (498h49m27.314860919s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444156695 (498h49m27.31523973s) [noise=true , address=77.90.31.16:4762 , piece_id=TO4OWX4NFCLFXXPW7NQWJ7NVXWG24TTBXN4ISBZE667U27FPTXQA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444525026 (498h49m27.315608101s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444959794 (498h49m27.31604284s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:28.898009100 (498h49m28.769092155s) [upload=437819]: Next returned nil segment Dec 26 10:18:28.898024930 (498h49m28.769107976s) [upload=437819]: Waiting for error group managing segments... Dec 26 10:18:29.317608319 (498h49m29.188691364s) [noise=true , address=145.79.227.52:4458 , piece_id=COMKCWPTVIBRPOPEWIDYU73EFQFXWVQY7IUFEM42HRQWXGJOUZ7A, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.417048480 (498h49m29.288131515s) [noise=true , address=64.49.13.137:4503 , piece_id=ZVP3DFOTW5GIW6WZ4KDQVV6GQV5MUS5VWLU5FDCAGFYFDDT2D5RA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.431472206 (498h49m29.302555241s) [noise=true , address=145.79.227.34:4915 , piece_id=ADW24WPJ5YJ73F5ABFTK3QSON3TBWUMZNCWNNBMV6ODFWNRFGX5A, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.531132256 (498h49m29.402215291s) [noise=true , address=77.90.31.14:4680 , piece_id=JK2NZRA5QXJJ2KWIQ3KPZT2GK2RJ3R6YGWVBLUHLGDYKBWUQKCIQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.539874212 (498h49m29.410957247s) [noise=true , address=45.91.116.12:4245 , piece_id=B5SSUXAQ7FF2WZKQGROIYQSZAMYR4M7K5WWA64O7KY46TEWWZQPQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.575613335 (498h49m29.44669638s) [noise=true , address=77.90.31.21:4883 , piece_id=45ESRA6YTW43MKEFIE2U3HPNW7JTPXM2SLNTHKY32F2YFORNVAQQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.588530461 (498h49m29.459613496s) [noise=true , address=77.90.31.17:4780 , piece_id=OXLKBKFLRXJFLRHIQ3ROAYBDCJUBZRIZ6U3RCZJOGK7Q7BXG26CQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.606622181 (498h49m29.477705216s) [noise=true , address=145.79.227.33:4876 , piece_id=OCV2FU5GXO5Q5TZ7CQ4WPGHSZ2MWFAFCP3VEX7TVGHELVNZJIGFA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.618087524 (498h49m29.489170559s) [noise=true , address=77.90.31.16:4762 , piece_id=TO4OWX4NFCLFXXPW7NQWJ7NVXWG24TTBXN4ISBZE667U27FPTXQA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.618804549 (498h49m29.489887594s) [noise=true , address=77.90.31.15:4730 , piece_id=RMDDZY2KUS6A4SLQW2B6JTHANDBCHGRWJIRHQQJ4X635I53B3ZHQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.627317289 (498h49m29.498400324s) [noise=true , address=77.90.31.18:4837 , piece_id=OC6GNBCZQMJCV7JSF7XHO3YHE7HMZ3HBWBKBYJRDKLMKMCYN2UFQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.743470775 (498h49m29.61455381s) [noise=true , address=77.90.31.19:4863 , piece_id=OQ7UPZNHXMG52A52UUIMOBDRI4B2EQE5VURHZM4PCOKYHVAG2C2Q, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.988678213 (498h49m29.859761248s) [noise=true , address=77.90.31.11:4560 , piece_id=G4EXQLB4I3GFTTLZNRTPTWRY2HUIBAMOTAAM333OJQB2SNZLQW7Q, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:30.173673986 (498h49m30.044757031s) [noise=true , address=145.79.227.44:4222 , piece_id=XU4TQPQXW4CXIJ3SI634Z32MMJB7GFQFHKNBZEGNHVONUQSVTSYQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:30.173691770 (498h49m30.044774815s) [seg_pos={0 0}, upload=437819]: Segment reached optimal threshold of 14 pieces. Dec 26 10:18:30.173872264 (498h49m30.044955299s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:30.173880039 (498h49m30.044963084s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174008883 (498h49m30.045091918s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection Dec 26 10:18:30.174017961 (498h49m30.045100996s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174049742 (498h49m30.045132777s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:30.174058829 (498h49m30.045141864s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174254883 (498h49m30.045337918s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jEyiUUU9NCVFMU8frNggxTTYBMkehsqN4GLsFRkrxubdobGY2): protocol: write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection; write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection Dec 26 10:18:30.174261886 (498h49m30.045344921s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174347448 (498h49m30.045430493s) [seg_pos={0 0}, upload=437819]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:18:30.174415378 (498h49m30.045498413s) [upload=437819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:30.174419666 (498h49m30.045502701s) [upload=437819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:30.472374192 (498h49m30.343457227s) [upload=437819]: Done uploading segments... Dec 26 10:18:30.472393109 (498h49m30.343476144s) [upload=437819]: Upload finished. err: <nil> Duration: 49.311132285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......] duration: 5.796537334s trace: https://jaeger.dev.storj.io/trace/27e7e2f289849627 FAIL (2025-12-26 10:22:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ) context deadline exceeded ======LOGS====== Dec 26 10:18:14.364246752 (498h49m14.235329787s) [upload=437816]: Starting upload Dec 26 10:18:14.364320452 (498h49m14.235403498s) [upload=437816]: Done starting upload Dec 26 10:18:14.364425312 (498h49m14.235508347s) [upload=437816]: Uploading segments... Dec 26 10:18:14.364586949 (498h49m14.235669994s) [upload=437816]: Got next segment. Inline: false Dec 26 10:18:14.364639129 (498h49m14.235722174s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:18:14.364645762 (498h49m14.235728827s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:18:14.609052935 (498h49m14.48013597s) [seg_pos={0 0}, upload=437816]: Begin upload segment... Dec 26 10:18:14.609112198 (498h49m14.480195243s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.609463647 (498h49m14.480546682s) [seg_pos={0 0}, upload=437816]: Done begin upload segment. Dec 26 10:18:14.609491941 (498h49m14.480574976s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610040396 (498h49m14.481123431s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610496985 (498h49m14.48158002s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610774714 (498h49m14.481857759s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611043024 (498h49m14.482126059s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611318639 (498h49m14.482401674s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611622909 (498h49m14.482705944s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.613638747 (498h49m14.484721782s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.614282773 (498h49m14.485365818s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.615846651 (498h49m14.486929696s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.617135205 (498h49m14.48821824s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.618487420 (498h49m14.489570465s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.619625988 (498h49m14.490709023s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.620824310 (498h49m14.491907345s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.622163480 (498h49m14.493246515s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:16.331618757 (498h49m16.202701802s) [upload=437816]: Next returned nil segment Dec 26 10:18:16.331635610 (498h49m16.202718645s) [upload=437816]: Waiting for error group managing segments... Dec 26 10:18:16.612650381 (498h49m16.483733416s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.622538299 (498h49m16.493621334s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.623481074 (498h49m16.494564119s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629239659 (498h49m16.500322704s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629900467 (498h49m16.500983522s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.632799878 (498h49m16.503882913s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.639602862 (498h49m16.510685897s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.644057315 (498h49m16.51514036s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.827488098 (498h49m16.698571143s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.933839800 (498h49m16.804922845s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.970575471 (498h49m16.841658526s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.046139899 (498h49m16.917222944s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615175711 (498h49m17.486258747s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615200027 (498h49m17.486283062s) [seg_pos={0 0}, upload=437816]: Segment reached optimal threshold of 13 pieces. Dec 26 10:18:17.615451246 (498h49m17.486534291s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615459532 (498h49m17.486542567s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615666807 (498h49m17.486749842s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615674000 (498h49m17.486757045s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615776666 (498h49m17.486859711s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615825538 (498h49m17.486908583s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615871687 (498h49m17.486954722s) [seg_pos={0 0}, upload=437816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:18:17.616073501 (498h49m17.487156556s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:17.616081616 (498h49m17.487164662s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:17.904101796 (498h49m17.775184841s) [upload=437816]: Done uploading segments... Dec 26 10:18:17.904125321 (498h49m17.775208356s) [upload=437816]: Upload finished. err: <nil> Duration: 5.796553836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X......X.........X..................X.......XXX....] duration: 41.463239151s trace: https://jaeger.dev.storj.io/trace/7aef545d37216976 FAIL (2025-12-26 10:22:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:4883 , piece_id=45ESRA6YTW43MKEFIE2U3HPNW7JTPXM2SLNTHKY32F2YFORNVAQQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.441584147 (498h49m27.312667192s) [noise=true , address=64.49.13.137:4503 , piece_id=ZVP3DFOTW5GIW6WZ4KDQVV6GQV5MUS5VWLU5FDCAGFYFDDT2D5RA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.441837960 (498h49m27.312921005s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442146617 (498h49m27.313229653s) [noise=true , address=77.90.31.11:4560 , piece_id=G4EXQLB4I3GFTTLZNRTPTWRY2HUIBAMOTAAM333OJQB2SNZLQW7Q, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442525168 (498h49m27.313608214s) [noise=true , address=145.79.227.52:4458 , piece_id=COMKCWPTVIBRPOPEWIDYU73EFQFXWVQY7IUFEM42HRQWXGJOUZ7A, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442608046 (498h49m27.313691091s) [noise=true , address=77.90.31.15:4730 , piece_id=RMDDZY2KUS6A4SLQW2B6JTHANDBCHGRWJIRHQQJ4X635I53B3ZHQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442903218 (498h49m27.313986264s) [noise=true , address=145.79.227.34:4915 , piece_id=ADW24WPJ5YJ73F5ABFTK3QSON3TBWUMZNCWNNBMV6ODFWNRFGX5A, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442912065 (498h49m27.3139951s) [noise=true , address=77.90.31.18:4837 , piece_id=OC6GNBCZQMJCV7JSF7XHO3YHE7HMZ3HBWBKBYJRDKLMKMCYN2UFQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443184915 (498h49m27.31426796s) [noise=true , address=77.90.31.14:4680 , piece_id=JK2NZRA5QXJJ2KWIQ3KPZT2GK2RJ3R6YGWVBLUHLGDYKBWUQKCIQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443385607 (498h49m27.314468652s) [noise=true , address=145.79.227.33:4876 , piece_id=OCV2FU5GXO5Q5TZ7CQ4WPGHSZ2MWFAFCP3VEX7TVGHELVNZJIGFA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443486720 (498h49m27.314569765s) [noise=true , address=45.91.116.12:4245 , piece_id=B5SSUXAQ7FF2WZKQGROIYQSZAMYR4M7K5WWA64O7KY46TEWWZQPQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443777884 (498h49m27.314860919s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444156695 (498h49m27.31523973s) [noise=true , address=77.90.31.16:4762 , piece_id=TO4OWX4NFCLFXXPW7NQWJ7NVXWG24TTBXN4ISBZE667U27FPTXQA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444525026 (498h49m27.315608101s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444959794 (498h49m27.31604284s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:28.898009100 (498h49m28.769092155s) [upload=437819]: Next returned nil segment Dec 26 10:18:28.898024930 (498h49m28.769107976s) [upload=437819]: Waiting for error group managing segments... Dec 26 10:18:29.317608319 (498h49m29.188691364s) [noise=true , address=145.79.227.52:4458 , piece_id=COMKCWPTVIBRPOPEWIDYU73EFQFXWVQY7IUFEM42HRQWXGJOUZ7A, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.417048480 (498h49m29.288131515s) [noise=true , address=64.49.13.137:4503 , piece_id=ZVP3DFOTW5GIW6WZ4KDQVV6GQV5MUS5VWLU5FDCAGFYFDDT2D5RA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.431472206 (498h49m29.302555241s) [noise=true , address=145.79.227.34:4915 , piece_id=ADW24WPJ5YJ73F5ABFTK3QSON3TBWUMZNCWNNBMV6ODFWNRFGX5A, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.531132256 (498h49m29.402215291s) [noise=true , address=77.90.31.14:4680 , piece_id=JK2NZRA5QXJJ2KWIQ3KPZT2GK2RJ3R6YGWVBLUHLGDYKBWUQKCIQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.539874212 (498h49m29.410957247s) [noise=true , address=45.91.116.12:4245 , piece_id=B5SSUXAQ7FF2WZKQGROIYQSZAMYR4M7K5WWA64O7KY46TEWWZQPQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.575613335 (498h49m29.44669638s) [noise=true , address=77.90.31.21:4883 , piece_id=45ESRA6YTW43MKEFIE2U3HPNW7JTPXM2SLNTHKY32F2YFORNVAQQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.588530461 (498h49m29.459613496s) [noise=true , address=77.90.31.17:4780 , piece_id=OXLKBKFLRXJFLRHIQ3ROAYBDCJUBZRIZ6U3RCZJOGK7Q7BXG26CQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.606622181 (498h49m29.477705216s) [noise=true , address=145.79.227.33:4876 , piece_id=OCV2FU5GXO5Q5TZ7CQ4WPGHSZ2MWFAFCP3VEX7TVGHELVNZJIGFA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.618087524 (498h49m29.489170559s) [noise=true , address=77.90.31.16:4762 , piece_id=TO4OWX4NFCLFXXPW7NQWJ7NVXWG24TTBXN4ISBZE667U27FPTXQA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.618804549 (498h49m29.489887594s) [noise=true , address=77.90.31.15:4730 , piece_id=RMDDZY2KUS6A4SLQW2B6JTHANDBCHGRWJIRHQQJ4X635I53B3ZHQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.627317289 (498h49m29.498400324s) [noise=true , address=77.90.31.18:4837 , piece_id=OC6GNBCZQMJCV7JSF7XHO3YHE7HMZ3HBWBKBYJRDKLMKMCYN2UFQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.743470775 (498h49m29.61455381s) [noise=true , address=77.90.31.19:4863 , piece_id=OQ7UPZNHXMG52A52UUIMOBDRI4B2EQE5VURHZM4PCOKYHVAG2C2Q, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.988678213 (498h49m29.859761248s) [noise=true , address=77.90.31.11:4560 , piece_id=G4EXQLB4I3GFTTLZNRTPTWRY2HUIBAMOTAAM333OJQB2SNZLQW7Q, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:30.173673986 (498h49m30.044757031s) [noise=true , address=145.79.227.44:4222 , piece_id=XU4TQPQXW4CXIJ3SI634Z32MMJB7GFQFHKNBZEGNHVONUQSVTSYQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:30.173691770 (498h49m30.044774815s) [seg_pos={0 0}, upload=437819]: Segment reached optimal threshold of 14 pieces. Dec 26 10:18:30.173872264 (498h49m30.044955299s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:30.173880039 (498h49m30.044963084s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174008883 (498h49m30.045091918s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection Dec 26 10:18:30.174017961 (498h49m30.045100996s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174049742 (498h49m30.045132777s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:30.174058829 (498h49m30.045141864s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174254883 (498h49m30.045337918s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jEyiUUU9NCVFMU8frNggxTTYBMkehsqN4GLsFRkrxubdobGY2): protocol: write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection; write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection Dec 26 10:18:30.174261886 (498h49m30.045344921s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174347448 (498h49m30.045430493s) [seg_pos={0 0}, upload=437819]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:18:30.174415378 (498h49m30.045498413s) [upload=437819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:30.174419666 (498h49m30.045502701s) [upload=437819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:30.472374192 (498h49m30.343457227s) [upload=437819]: Done uploading segments... Dec 26 10:18:30.472393109 (498h49m30.343476144s) [upload=437819]: Upload finished. err: <nil> Duration: 41.463263977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX......] duration: 5.708382901s trace: https://jaeger.dev.storj.io/trace/27e7e2f289849627 FAIL (2025-12-26 10:22:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ) context deadline exceeded ======LOGS====== Dec 26 10:18:14.364246752 (498h49m14.235329787s) [upload=437816]: Starting upload Dec 26 10:18:14.364320452 (498h49m14.235403498s) [upload=437816]: Done starting upload Dec 26 10:18:14.364425312 (498h49m14.235508347s) [upload=437816]: Uploading segments... Dec 26 10:18:14.364586949 (498h49m14.235669994s) [upload=437816]: Got next segment. Inline: false Dec 26 10:18:14.364639129 (498h49m14.235722174s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:18:14.364645762 (498h49m14.235728827s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:18:14.609052935 (498h49m14.48013597s) [seg_pos={0 0}, upload=437816]: Begin upload segment... Dec 26 10:18:14.609112198 (498h49m14.480195243s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.609463647 (498h49m14.480546682s) [seg_pos={0 0}, upload=437816]: Done begin upload segment. Dec 26 10:18:14.609491941 (498h49m14.480574976s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610040396 (498h49m14.481123431s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610496985 (498h49m14.48158002s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610774714 (498h49m14.481857759s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611043024 (498h49m14.482126059s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611318639 (498h49m14.482401674s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611622909 (498h49m14.482705944s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.613638747 (498h49m14.484721782s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.614282773 (498h49m14.485365818s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.615846651 (498h49m14.486929696s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.617135205 (498h49m14.48821824s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.618487420 (498h49m14.489570465s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.619625988 (498h49m14.490709023s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.620824310 (498h49m14.491907345s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.622163480 (498h49m14.493246515s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:16.331618757 (498h49m16.202701802s) [upload=437816]: Next returned nil segment Dec 26 10:18:16.331635610 (498h49m16.202718645s) [upload=437816]: Waiting for error group managing segments... Dec 26 10:18:16.612650381 (498h49m16.483733416s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.622538299 (498h49m16.493621334s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.623481074 (498h49m16.494564119s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629239659 (498h49m16.500322704s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629900467 (498h49m16.500983522s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.632799878 (498h49m16.503882913s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.639602862 (498h49m16.510685897s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.644057315 (498h49m16.51514036s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.827488098 (498h49m16.698571143s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.933839800 (498h49m16.804922845s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.970575471 (498h49m16.841658526s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.046139899 (498h49m16.917222944s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615175711 (498h49m17.486258747s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615200027 (498h49m17.486283062s) [seg_pos={0 0}, upload=437816]: Segment reached optimal threshold of 13 pieces. Dec 26 10:18:17.615451246 (498h49m17.486534291s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615459532 (498h49m17.486542567s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615666807 (498h49m17.486749842s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615674000 (498h49m17.486757045s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615776666 (498h49m17.486859711s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615825538 (498h49m17.486908583s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615871687 (498h49m17.486954722s) [seg_pos={0 0}, upload=437816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:18:17.616073501 (498h49m17.487156556s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:17.616081616 (498h49m17.487164662s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:17.904101796 (498h49m17.775184841s) [upload=437816]: Done uploading segments... Dec 26 10:18:17.904125321 (498h49m17.775208356s) [upload=437816]: Upload finished. err: <nil> Duration: 5.708413968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X......X.........X..................X.......XXX...] duration: 32.984054026s trace: https://jaeger.dev.storj.io/trace/7aef545d37216976 FAIL (2025-12-26 10:21:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:4883 , piece_id=45ESRA6YTW43MKEFIE2U3HPNW7JTPXM2SLNTHKY32F2YFORNVAQQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.441584147 (498h49m27.312667192s) [noise=true , address=64.49.13.137:4503 , piece_id=ZVP3DFOTW5GIW6WZ4KDQVV6GQV5MUS5VWLU5FDCAGFYFDDT2D5RA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.441837960 (498h49m27.312921005s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442146617 (498h49m27.313229653s) [noise=true , address=77.90.31.11:4560 , piece_id=G4EXQLB4I3GFTTLZNRTPTWRY2HUIBAMOTAAM333OJQB2SNZLQW7Q, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442525168 (498h49m27.313608214s) [noise=true , address=145.79.227.52:4458 , piece_id=COMKCWPTVIBRPOPEWIDYU73EFQFXWVQY7IUFEM42HRQWXGJOUZ7A, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442608046 (498h49m27.313691091s) [noise=true , address=77.90.31.15:4730 , piece_id=RMDDZY2KUS6A4SLQW2B6JTHANDBCHGRWJIRHQQJ4X635I53B3ZHQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442903218 (498h49m27.313986264s) [noise=true , address=145.79.227.34:4915 , piece_id=ADW24WPJ5YJ73F5ABFTK3QSON3TBWUMZNCWNNBMV6ODFWNRFGX5A, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442912065 (498h49m27.3139951s) [noise=true , address=77.90.31.18:4837 , piece_id=OC6GNBCZQMJCV7JSF7XHO3YHE7HMZ3HBWBKBYJRDKLMKMCYN2UFQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443184915 (498h49m27.31426796s) [noise=true , address=77.90.31.14:4680 , piece_id=JK2NZRA5QXJJ2KWIQ3KPZT2GK2RJ3R6YGWVBLUHLGDYKBWUQKCIQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443385607 (498h49m27.314468652s) [noise=true , address=145.79.227.33:4876 , piece_id=OCV2FU5GXO5Q5TZ7CQ4WPGHSZ2MWFAFCP3VEX7TVGHELVNZJIGFA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443486720 (498h49m27.314569765s) [noise=true , address=45.91.116.12:4245 , piece_id=B5SSUXAQ7FF2WZKQGROIYQSZAMYR4M7K5WWA64O7KY46TEWWZQPQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443777884 (498h49m27.314860919s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444156695 (498h49m27.31523973s) [noise=true , address=77.90.31.16:4762 , piece_id=TO4OWX4NFCLFXXPW7NQWJ7NVXWG24TTBXN4ISBZE667U27FPTXQA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444525026 (498h49m27.315608101s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444959794 (498h49m27.31604284s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:28.898009100 (498h49m28.769092155s) [upload=437819]: Next returned nil segment Dec 26 10:18:28.898024930 (498h49m28.769107976s) [upload=437819]: Waiting for error group managing segments... Dec 26 10:18:29.317608319 (498h49m29.188691364s) [noise=true , address=145.79.227.52:4458 , piece_id=COMKCWPTVIBRPOPEWIDYU73EFQFXWVQY7IUFEM42HRQWXGJOUZ7A, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.417048480 (498h49m29.288131515s) [noise=true , address=64.49.13.137:4503 , piece_id=ZVP3DFOTW5GIW6WZ4KDQVV6GQV5MUS5VWLU5FDCAGFYFDDT2D5RA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.431472206 (498h49m29.302555241s) [noise=true , address=145.79.227.34:4915 , piece_id=ADW24WPJ5YJ73F5ABFTK3QSON3TBWUMZNCWNNBMV6ODFWNRFGX5A, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.531132256 (498h49m29.402215291s) [noise=true , address=77.90.31.14:4680 , piece_id=JK2NZRA5QXJJ2KWIQ3KPZT2GK2RJ3R6YGWVBLUHLGDYKBWUQKCIQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.539874212 (498h49m29.410957247s) [noise=true , address=45.91.116.12:4245 , piece_id=B5SSUXAQ7FF2WZKQGROIYQSZAMYR4M7K5WWA64O7KY46TEWWZQPQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.575613335 (498h49m29.44669638s) [noise=true , address=77.90.31.21:4883 , piece_id=45ESRA6YTW43MKEFIE2U3HPNW7JTPXM2SLNTHKY32F2YFORNVAQQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.588530461 (498h49m29.459613496s) [noise=true , address=77.90.31.17:4780 , piece_id=OXLKBKFLRXJFLRHIQ3ROAYBDCJUBZRIZ6U3RCZJOGK7Q7BXG26CQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.606622181 (498h49m29.477705216s) [noise=true , address=145.79.227.33:4876 , piece_id=OCV2FU5GXO5Q5TZ7CQ4WPGHSZ2MWFAFCP3VEX7TVGHELVNZJIGFA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.618087524 (498h49m29.489170559s) [noise=true , address=77.90.31.16:4762 , piece_id=TO4OWX4NFCLFXXPW7NQWJ7NVXWG24TTBXN4ISBZE667U27FPTXQA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.618804549 (498h49m29.489887594s) [noise=true , address=77.90.31.15:4730 , piece_id=RMDDZY2KUS6A4SLQW2B6JTHANDBCHGRWJIRHQQJ4X635I53B3ZHQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.627317289 (498h49m29.498400324s) [noise=true , address=77.90.31.18:4837 , piece_id=OC6GNBCZQMJCV7JSF7XHO3YHE7HMZ3HBWBKBYJRDKLMKMCYN2UFQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.743470775 (498h49m29.61455381s) [noise=true , address=77.90.31.19:4863 , piece_id=OQ7UPZNHXMG52A52UUIMOBDRI4B2EQE5VURHZM4PCOKYHVAG2C2Q, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.988678213 (498h49m29.859761248s) [noise=true , address=77.90.31.11:4560 , piece_id=G4EXQLB4I3GFTTLZNRTPTWRY2HUIBAMOTAAM333OJQB2SNZLQW7Q, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:30.173673986 (498h49m30.044757031s) [noise=true , address=145.79.227.44:4222 , piece_id=XU4TQPQXW4CXIJ3SI634Z32MMJB7GFQFHKNBZEGNHVONUQSVTSYQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:30.173691770 (498h49m30.044774815s) [seg_pos={0 0}, upload=437819]: Segment reached optimal threshold of 14 pieces. Dec 26 10:18:30.173872264 (498h49m30.044955299s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:30.173880039 (498h49m30.044963084s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174008883 (498h49m30.045091918s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection Dec 26 10:18:30.174017961 (498h49m30.045100996s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174049742 (498h49m30.045132777s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:30.174058829 (498h49m30.045141864s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174254883 (498h49m30.045337918s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jEyiUUU9NCVFMU8frNggxTTYBMkehsqN4GLsFRkrxubdobGY2): protocol: write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection; write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection Dec 26 10:18:30.174261886 (498h49m30.045344921s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174347448 (498h49m30.045430493s) [seg_pos={0 0}, upload=437819]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:18:30.174415378 (498h49m30.045498413s) [upload=437819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:30.174419666 (498h49m30.045502701s) [upload=437819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:30.472374192 (498h49m30.343457227s) [upload=437819]: Done uploading segments... Dec 26 10:18:30.472393109 (498h49m30.343476144s) [upload=437819]: Upload finished. err: <nil> Duration: 32.984072651s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.....] duration: 5.692146224s trace: https://jaeger.dev.storj.io/trace/27e7e2f289849627 FAIL (2025-12-26 10:21:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ) context deadline exceeded ======LOGS====== Dec 26 10:18:14.364246752 (498h49m14.235329787s) [upload=437816]: Starting upload Dec 26 10:18:14.364320452 (498h49m14.235403498s) [upload=437816]: Done starting upload Dec 26 10:18:14.364425312 (498h49m14.235508347s) [upload=437816]: Uploading segments... Dec 26 10:18:14.364586949 (498h49m14.235669994s) [upload=437816]: Got next segment. Inline: false Dec 26 10:18:14.364639129 (498h49m14.235722174s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:18:14.364645762 (498h49m14.235728827s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:18:14.609052935 (498h49m14.48013597s) [seg_pos={0 0}, upload=437816]: Begin upload segment... Dec 26 10:18:14.609112198 (498h49m14.480195243s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.609463647 (498h49m14.480546682s) [seg_pos={0 0}, upload=437816]: Done begin upload segment. Dec 26 10:18:14.609491941 (498h49m14.480574976s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610040396 (498h49m14.481123431s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610496985 (498h49m14.48158002s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610774714 (498h49m14.481857759s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611043024 (498h49m14.482126059s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611318639 (498h49m14.482401674s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611622909 (498h49m14.482705944s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.613638747 (498h49m14.484721782s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.614282773 (498h49m14.485365818s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.615846651 (498h49m14.486929696s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.617135205 (498h49m14.48821824s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.618487420 (498h49m14.489570465s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.619625988 (498h49m14.490709023s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.620824310 (498h49m14.491907345s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.622163480 (498h49m14.493246515s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:16.331618757 (498h49m16.202701802s) [upload=437816]: Next returned nil segment Dec 26 10:18:16.331635610 (498h49m16.202718645s) [upload=437816]: Waiting for error group managing segments... Dec 26 10:18:16.612650381 (498h49m16.483733416s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.622538299 (498h49m16.493621334s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.623481074 (498h49m16.494564119s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629239659 (498h49m16.500322704s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629900467 (498h49m16.500983522s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.632799878 (498h49m16.503882913s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.639602862 (498h49m16.510685897s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.644057315 (498h49m16.51514036s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.827488098 (498h49m16.698571143s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.933839800 (498h49m16.804922845s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.970575471 (498h49m16.841658526s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.046139899 (498h49m16.917222944s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615175711 (498h49m17.486258747s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615200027 (498h49m17.486283062s) [seg_pos={0 0}, upload=437816]: Segment reached optimal threshold of 13 pieces. Dec 26 10:18:17.615451246 (498h49m17.486534291s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615459532 (498h49m17.486542567s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615666807 (498h49m17.486749842s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615674000 (498h49m17.486757045s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615776666 (498h49m17.486859711s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615825538 (498h49m17.486908583s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615871687 (498h49m17.486954722s) [seg_pos={0 0}, upload=437816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:18:17.616073501 (498h49m17.487156556s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:17.616081616 (498h49m17.487164662s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:17.904101796 (498h49m17.775184841s) [upload=437816]: Done uploading segments... Dec 26 10:18:17.904125321 (498h49m17.775208356s) [upload=437816]: Upload finished. err: <nil> Duration: 5.692172696s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X......X.........X..................X.......XXX..] duration: 51.116972432s trace: https://jaeger.dev.storj.io/trace/7aef545d37216976 FAIL (2025-12-26 10:21:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:4883 , piece_id=45ESRA6YTW43MKEFIE2U3HPNW7JTPXM2SLNTHKY32F2YFORNVAQQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.441584147 (498h49m27.312667192s) [noise=true , address=64.49.13.137:4503 , piece_id=ZVP3DFOTW5GIW6WZ4KDQVV6GQV5MUS5VWLU5FDCAGFYFDDT2D5RA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.441837960 (498h49m27.312921005s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442146617 (498h49m27.313229653s) [noise=true , address=77.90.31.11:4560 , piece_id=G4EXQLB4I3GFTTLZNRTPTWRY2HUIBAMOTAAM333OJQB2SNZLQW7Q, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442525168 (498h49m27.313608214s) [noise=true , address=145.79.227.52:4458 , piece_id=COMKCWPTVIBRPOPEWIDYU73EFQFXWVQY7IUFEM42HRQWXGJOUZ7A, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442608046 (498h49m27.313691091s) [noise=true , address=77.90.31.15:4730 , piece_id=RMDDZY2KUS6A4SLQW2B6JTHANDBCHGRWJIRHQQJ4X635I53B3ZHQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442903218 (498h49m27.313986264s) [noise=true , address=145.79.227.34:4915 , piece_id=ADW24WPJ5YJ73F5ABFTK3QSON3TBWUMZNCWNNBMV6ODFWNRFGX5A, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442912065 (498h49m27.3139951s) [noise=true , address=77.90.31.18:4837 , piece_id=OC6GNBCZQMJCV7JSF7XHO3YHE7HMZ3HBWBKBYJRDKLMKMCYN2UFQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443184915 (498h49m27.31426796s) [noise=true , address=77.90.31.14:4680 , piece_id=JK2NZRA5QXJJ2KWIQ3KPZT2GK2RJ3R6YGWVBLUHLGDYKBWUQKCIQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443385607 (498h49m27.314468652s) [noise=true , address=145.79.227.33:4876 , piece_id=OCV2FU5GXO5Q5TZ7CQ4WPGHSZ2MWFAFCP3VEX7TVGHELVNZJIGFA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443486720 (498h49m27.314569765s) [noise=true , address=45.91.116.12:4245 , piece_id=B5SSUXAQ7FF2WZKQGROIYQSZAMYR4M7K5WWA64O7KY46TEWWZQPQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443777884 (498h49m27.314860919s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444156695 (498h49m27.31523973s) [noise=true , address=77.90.31.16:4762 , piece_id=TO4OWX4NFCLFXXPW7NQWJ7NVXWG24TTBXN4ISBZE667U27FPTXQA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444525026 (498h49m27.315608101s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444959794 (498h49m27.31604284s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:28.898009100 (498h49m28.769092155s) [upload=437819]: Next returned nil segment Dec 26 10:18:28.898024930 (498h49m28.769107976s) [upload=437819]: Waiting for error group managing segments... Dec 26 10:18:29.317608319 (498h49m29.188691364s) [noise=true , address=145.79.227.52:4458 , piece_id=COMKCWPTVIBRPOPEWIDYU73EFQFXWVQY7IUFEM42HRQWXGJOUZ7A, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.417048480 (498h49m29.288131515s) [noise=true , address=64.49.13.137:4503 , piece_id=ZVP3DFOTW5GIW6WZ4KDQVV6GQV5MUS5VWLU5FDCAGFYFDDT2D5RA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.431472206 (498h49m29.302555241s) [noise=true , address=145.79.227.34:4915 , piece_id=ADW24WPJ5YJ73F5ABFTK3QSON3TBWUMZNCWNNBMV6ODFWNRFGX5A, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.531132256 (498h49m29.402215291s) [noise=true , address=77.90.31.14:4680 , piece_id=JK2NZRA5QXJJ2KWIQ3KPZT2GK2RJ3R6YGWVBLUHLGDYKBWUQKCIQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.539874212 (498h49m29.410957247s) [noise=true , address=45.91.116.12:4245 , piece_id=B5SSUXAQ7FF2WZKQGROIYQSZAMYR4M7K5WWA64O7KY46TEWWZQPQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.575613335 (498h49m29.44669638s) [noise=true , address=77.90.31.21:4883 , piece_id=45ESRA6YTW43MKEFIE2U3HPNW7JTPXM2SLNTHKY32F2YFORNVAQQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.588530461 (498h49m29.459613496s) [noise=true , address=77.90.31.17:4780 , piece_id=OXLKBKFLRXJFLRHIQ3ROAYBDCJUBZRIZ6U3RCZJOGK7Q7BXG26CQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.606622181 (498h49m29.477705216s) [noise=true , address=145.79.227.33:4876 , piece_id=OCV2FU5GXO5Q5TZ7CQ4WPGHSZ2MWFAFCP3VEX7TVGHELVNZJIGFA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.618087524 (498h49m29.489170559s) [noise=true , address=77.90.31.16:4762 , piece_id=TO4OWX4NFCLFXXPW7NQWJ7NVXWG24TTBXN4ISBZE667U27FPTXQA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.618804549 (498h49m29.489887594s) [noise=true , address=77.90.31.15:4730 , piece_id=RMDDZY2KUS6A4SLQW2B6JTHANDBCHGRWJIRHQQJ4X635I53B3ZHQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.627317289 (498h49m29.498400324s) [noise=true , address=77.90.31.18:4837 , piece_id=OC6GNBCZQMJCV7JSF7XHO3YHE7HMZ3HBWBKBYJRDKLMKMCYN2UFQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.743470775 (498h49m29.61455381s) [noise=true , address=77.90.31.19:4863 , piece_id=OQ7UPZNHXMG52A52UUIMOBDRI4B2EQE5VURHZM4PCOKYHVAG2C2Q, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.988678213 (498h49m29.859761248s) [noise=true , address=77.90.31.11:4560 , piece_id=G4EXQLB4I3GFTTLZNRTPTWRY2HUIBAMOTAAM333OJQB2SNZLQW7Q, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:30.173673986 (498h49m30.044757031s) [noise=true , address=145.79.227.44:4222 , piece_id=XU4TQPQXW4CXIJ3SI634Z32MMJB7GFQFHKNBZEGNHVONUQSVTSYQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:30.173691770 (498h49m30.044774815s) [seg_pos={0 0}, upload=437819]: Segment reached optimal threshold of 14 pieces. Dec 26 10:18:30.173872264 (498h49m30.044955299s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:30.173880039 (498h49m30.044963084s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174008883 (498h49m30.045091918s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection Dec 26 10:18:30.174017961 (498h49m30.045100996s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174049742 (498h49m30.045132777s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:30.174058829 (498h49m30.045141864s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174254883 (498h49m30.045337918s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jEyiUUU9NCVFMU8frNggxTTYBMkehsqN4GLsFRkrxubdobGY2): protocol: write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection; write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection Dec 26 10:18:30.174261886 (498h49m30.045344921s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174347448 (498h49m30.045430493s) [seg_pos={0 0}, upload=437819]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:18:30.174415378 (498h49m30.045498413s) [upload=437819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:30.174419666 (498h49m30.045502701s) [upload=437819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:30.472374192 (498h49m30.343457227s) [upload=437819]: Done uploading segments... Dec 26 10:18:30.472393109 (498h49m30.343476144s) [upload=437819]: Upload finished. err: <nil> Duration: 51.116995316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX....] duration: 5.922631727s trace: https://jaeger.dev.storj.io/trace/27e7e2f289849627 FAIL (2025-12-26 10:20:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ) context deadline exceeded ======LOGS====== Dec 26 10:18:14.364246752 (498h49m14.235329787s) [upload=437816]: Starting upload Dec 26 10:18:14.364320452 (498h49m14.235403498s) [upload=437816]: Done starting upload Dec 26 10:18:14.364425312 (498h49m14.235508347s) [upload=437816]: Uploading segments... Dec 26 10:18:14.364586949 (498h49m14.235669994s) [upload=437816]: Got next segment. Inline: false Dec 26 10:18:14.364639129 (498h49m14.235722174s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:18:14.364645762 (498h49m14.235728827s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:18:14.609052935 (498h49m14.48013597s) [seg_pos={0 0}, upload=437816]: Begin upload segment... Dec 26 10:18:14.609112198 (498h49m14.480195243s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.609463647 (498h49m14.480546682s) [seg_pos={0 0}, upload=437816]: Done begin upload segment. Dec 26 10:18:14.609491941 (498h49m14.480574976s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610040396 (498h49m14.481123431s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610496985 (498h49m14.48158002s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610774714 (498h49m14.481857759s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611043024 (498h49m14.482126059s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611318639 (498h49m14.482401674s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611622909 (498h49m14.482705944s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.613638747 (498h49m14.484721782s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.614282773 (498h49m14.485365818s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.615846651 (498h49m14.486929696s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.617135205 (498h49m14.48821824s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.618487420 (498h49m14.489570465s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.619625988 (498h49m14.490709023s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.620824310 (498h49m14.491907345s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.622163480 (498h49m14.493246515s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:16.331618757 (498h49m16.202701802s) [upload=437816]: Next returned nil segment Dec 26 10:18:16.331635610 (498h49m16.202718645s) [upload=437816]: Waiting for error group managing segments... Dec 26 10:18:16.612650381 (498h49m16.483733416s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.622538299 (498h49m16.493621334s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.623481074 (498h49m16.494564119s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629239659 (498h49m16.500322704s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629900467 (498h49m16.500983522s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.632799878 (498h49m16.503882913s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.639602862 (498h49m16.510685897s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.644057315 (498h49m16.51514036s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.827488098 (498h49m16.698571143s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.933839800 (498h49m16.804922845s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.970575471 (498h49m16.841658526s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.046139899 (498h49m16.917222944s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615175711 (498h49m17.486258747s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615200027 (498h49m17.486283062s) [seg_pos={0 0}, upload=437816]: Segment reached optimal threshold of 13 pieces. Dec 26 10:18:17.615451246 (498h49m17.486534291s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615459532 (498h49m17.486542567s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615666807 (498h49m17.486749842s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615674000 (498h49m17.486757045s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615776666 (498h49m17.486859711s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615825538 (498h49m17.486908583s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615871687 (498h49m17.486954722s) [seg_pos={0 0}, upload=437816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:18:17.616073501 (498h49m17.487156556s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:17.616081616 (498h49m17.487164662s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:17.904101796 (498h49m17.775184841s) [upload=437816]: Done uploading segments... Dec 26 10:18:17.904125321 (498h49m17.775208356s) [upload=437816]: Upload finished. err: <nil> Duration: 5.922657136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X......X.........X..................X.......XXX.] duration: 53.551793498s trace: https://jaeger.dev.storj.io/trace/7aef545d37216976 FAIL (2025-12-26 10:20:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:4883 , piece_id=45ESRA6YTW43MKEFIE2U3HPNW7JTPXM2SLNTHKY32F2YFORNVAQQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.441584147 (498h49m27.312667192s) [noise=true , address=64.49.13.137:4503 , piece_id=ZVP3DFOTW5GIW6WZ4KDQVV6GQV5MUS5VWLU5FDCAGFYFDDT2D5RA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.441837960 (498h49m27.312921005s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442146617 (498h49m27.313229653s) [noise=true , address=77.90.31.11:4560 , piece_id=G4EXQLB4I3GFTTLZNRTPTWRY2HUIBAMOTAAM333OJQB2SNZLQW7Q, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442525168 (498h49m27.313608214s) [noise=true , address=145.79.227.52:4458 , piece_id=COMKCWPTVIBRPOPEWIDYU73EFQFXWVQY7IUFEM42HRQWXGJOUZ7A, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442608046 (498h49m27.313691091s) [noise=true , address=77.90.31.15:4730 , piece_id=RMDDZY2KUS6A4SLQW2B6JTHANDBCHGRWJIRHQQJ4X635I53B3ZHQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442903218 (498h49m27.313986264s) [noise=true , address=145.79.227.34:4915 , piece_id=ADW24WPJ5YJ73F5ABFTK3QSON3TBWUMZNCWNNBMV6ODFWNRFGX5A, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442912065 (498h49m27.3139951s) [noise=true , address=77.90.31.18:4837 , piece_id=OC6GNBCZQMJCV7JSF7XHO3YHE7HMZ3HBWBKBYJRDKLMKMCYN2UFQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443184915 (498h49m27.31426796s) [noise=true , address=77.90.31.14:4680 , piece_id=JK2NZRA5QXJJ2KWIQ3KPZT2GK2RJ3R6YGWVBLUHLGDYKBWUQKCIQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443385607 (498h49m27.314468652s) [noise=true , address=145.79.227.33:4876 , piece_id=OCV2FU5GXO5Q5TZ7CQ4WPGHSZ2MWFAFCP3VEX7TVGHELVNZJIGFA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443486720 (498h49m27.314569765s) [noise=true , address=45.91.116.12:4245 , piece_id=B5SSUXAQ7FF2WZKQGROIYQSZAMYR4M7K5WWA64O7KY46TEWWZQPQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443777884 (498h49m27.314860919s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444156695 (498h49m27.31523973s) [noise=true , address=77.90.31.16:4762 , piece_id=TO4OWX4NFCLFXXPW7NQWJ7NVXWG24TTBXN4ISBZE667U27FPTXQA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444525026 (498h49m27.315608101s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444959794 (498h49m27.31604284s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:28.898009100 (498h49m28.769092155s) [upload=437819]: Next returned nil segment Dec 26 10:18:28.898024930 (498h49m28.769107976s) [upload=437819]: Waiting for error group managing segments... Dec 26 10:18:29.317608319 (498h49m29.188691364s) [noise=true , address=145.79.227.52:4458 , piece_id=COMKCWPTVIBRPOPEWIDYU73EFQFXWVQY7IUFEM42HRQWXGJOUZ7A, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.417048480 (498h49m29.288131515s) [noise=true , address=64.49.13.137:4503 , piece_id=ZVP3DFOTW5GIW6WZ4KDQVV6GQV5MUS5VWLU5FDCAGFYFDDT2D5RA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.431472206 (498h49m29.302555241s) [noise=true , address=145.79.227.34:4915 , piece_id=ADW24WPJ5YJ73F5ABFTK3QSON3TBWUMZNCWNNBMV6ODFWNRFGX5A, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.531132256 (498h49m29.402215291s) [noise=true , address=77.90.31.14:4680 , piece_id=JK2NZRA5QXJJ2KWIQ3KPZT2GK2RJ3R6YGWVBLUHLGDYKBWUQKCIQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.539874212 (498h49m29.410957247s) [noise=true , address=45.91.116.12:4245 , piece_id=B5SSUXAQ7FF2WZKQGROIYQSZAMYR4M7K5WWA64O7KY46TEWWZQPQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.575613335 (498h49m29.44669638s) [noise=true , address=77.90.31.21:4883 , piece_id=45ESRA6YTW43MKEFIE2U3HPNW7JTPXM2SLNTHKY32F2YFORNVAQQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.588530461 (498h49m29.459613496s) [noise=true , address=77.90.31.17:4780 , piece_id=OXLKBKFLRXJFLRHIQ3ROAYBDCJUBZRIZ6U3RCZJOGK7Q7BXG26CQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.606622181 (498h49m29.477705216s) [noise=true , address=145.79.227.33:4876 , piece_id=OCV2FU5GXO5Q5TZ7CQ4WPGHSZ2MWFAFCP3VEX7TVGHELVNZJIGFA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.618087524 (498h49m29.489170559s) [noise=true , address=77.90.31.16:4762 , piece_id=TO4OWX4NFCLFXXPW7NQWJ7NVXWG24TTBXN4ISBZE667U27FPTXQA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.618804549 (498h49m29.489887594s) [noise=true , address=77.90.31.15:4730 , piece_id=RMDDZY2KUS6A4SLQW2B6JTHANDBCHGRWJIRHQQJ4X635I53B3ZHQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.627317289 (498h49m29.498400324s) [noise=true , address=77.90.31.18:4837 , piece_id=OC6GNBCZQMJCV7JSF7XHO3YHE7HMZ3HBWBKBYJRDKLMKMCYN2UFQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.743470775 (498h49m29.61455381s) [noise=true , address=77.90.31.19:4863 , piece_id=OQ7UPZNHXMG52A52UUIMOBDRI4B2EQE5VURHZM4PCOKYHVAG2C2Q, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.988678213 (498h49m29.859761248s) [noise=true , address=77.90.31.11:4560 , piece_id=G4EXQLB4I3GFTTLZNRTPTWRY2HUIBAMOTAAM333OJQB2SNZLQW7Q, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:30.173673986 (498h49m30.044757031s) [noise=true , address=145.79.227.44:4222 , piece_id=XU4TQPQXW4CXIJ3SI634Z32MMJB7GFQFHKNBZEGNHVONUQSVTSYQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:30.173691770 (498h49m30.044774815s) [seg_pos={0 0}, upload=437819]: Segment reached optimal threshold of 14 pieces. Dec 26 10:18:30.173872264 (498h49m30.044955299s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:30.173880039 (498h49m30.044963084s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174008883 (498h49m30.045091918s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection Dec 26 10:18:30.174017961 (498h49m30.045100996s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174049742 (498h49m30.045132777s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:30.174058829 (498h49m30.045141864s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174254883 (498h49m30.045337918s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jEyiUUU9NCVFMU8frNggxTTYBMkehsqN4GLsFRkrxubdobGY2): protocol: write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection; write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection Dec 26 10:18:30.174261886 (498h49m30.045344921s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174347448 (498h49m30.045430493s) [seg_pos={0 0}, upload=437819]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:18:30.174415378 (498h49m30.045498413s) [upload=437819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:30.174419666 (498h49m30.045502701s) [upload=437819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:30.472374192 (498h49m30.343457227s) [upload=437819]: Done uploading segments... Dec 26 10:18:30.472393109 (498h49m30.343476144s) [upload=437819]: Upload finished. err: <nil> Duration: 53.551809678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [..X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX...] duration: 6.159590657s trace: https://jaeger.dev.storj.io/trace/27e7e2f289849627 FAIL (2025-12-26 10:20:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ) context deadline exceeded ======LOGS====== Dec 26 10:18:14.364246752 (498h49m14.235329787s) [upload=437816]: Starting upload Dec 26 10:18:14.364320452 (498h49m14.235403498s) [upload=437816]: Done starting upload Dec 26 10:18:14.364425312 (498h49m14.235508347s) [upload=437816]: Uploading segments... Dec 26 10:18:14.364586949 (498h49m14.235669994s) [upload=437816]: Got next segment. Inline: false Dec 26 10:18:14.364639129 (498h49m14.235722174s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:18:14.364645762 (498h49m14.235728827s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:18:14.609052935 (498h49m14.48013597s) [seg_pos={0 0}, upload=437816]: Begin upload segment... Dec 26 10:18:14.609112198 (498h49m14.480195243s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.609463647 (498h49m14.480546682s) [seg_pos={0 0}, upload=437816]: Done begin upload segment. Dec 26 10:18:14.609491941 (498h49m14.480574976s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610040396 (498h49m14.481123431s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610496985 (498h49m14.48158002s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610774714 (498h49m14.481857759s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611043024 (498h49m14.482126059s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611318639 (498h49m14.482401674s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611622909 (498h49m14.482705944s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.613638747 (498h49m14.484721782s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.614282773 (498h49m14.485365818s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.615846651 (498h49m14.486929696s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.617135205 (498h49m14.48821824s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.618487420 (498h49m14.489570465s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.619625988 (498h49m14.490709023s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.620824310 (498h49m14.491907345s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.622163480 (498h49m14.493246515s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:16.331618757 (498h49m16.202701802s) [upload=437816]: Next returned nil segment Dec 26 10:18:16.331635610 (498h49m16.202718645s) [upload=437816]: Waiting for error group managing segments... Dec 26 10:18:16.612650381 (498h49m16.483733416s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.622538299 (498h49m16.493621334s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.623481074 (498h49m16.494564119s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629239659 (498h49m16.500322704s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629900467 (498h49m16.500983522s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.632799878 (498h49m16.503882913s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.639602862 (498h49m16.510685897s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.644057315 (498h49m16.51514036s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.827488098 (498h49m16.698571143s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.933839800 (498h49m16.804922845s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.970575471 (498h49m16.841658526s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.046139899 (498h49m16.917222944s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615175711 (498h49m17.486258747s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615200027 (498h49m17.486283062s) [seg_pos={0 0}, upload=437816]: Segment reached optimal threshold of 13 pieces. Dec 26 10:18:17.615451246 (498h49m17.486534291s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615459532 (498h49m17.486542567s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615666807 (498h49m17.486749842s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615674000 (498h49m17.486757045s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615776666 (498h49m17.486859711s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615825538 (498h49m17.486908583s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615871687 (498h49m17.486954722s) [seg_pos={0 0}, upload=437816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:18:17.616073501 (498h49m17.487156556s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:17.616081616 (498h49m17.487164662s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:17.904101796 (498h49m17.775184841s) [upload=437816]: Done uploading segments... Dec 26 10:18:17.904125321 (498h49m17.775208356s) [upload=437816]: Upload finished. err: <nil> Duration: 6.159621866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [...X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX..] duration: 6.009451436s trace: https://jaeger.dev.storj.io/trace/27e7e2f289849627 FAIL (2025-12-26 10:20:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ) context deadline exceeded ======LOGS====== Dec 26 10:18:14.364246752 (498h49m14.235329787s) [upload=437816]: Starting upload Dec 26 10:18:14.364320452 (498h49m14.235403498s) [upload=437816]: Done starting upload Dec 26 10:18:14.364425312 (498h49m14.235508347s) [upload=437816]: Uploading segments... Dec 26 10:18:14.364586949 (498h49m14.235669994s) [upload=437816]: Got next segment. Inline: false Dec 26 10:18:14.364639129 (498h49m14.235722174s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:18:14.364645762 (498h49m14.235728827s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:18:14.609052935 (498h49m14.48013597s) [seg_pos={0 0}, upload=437816]: Begin upload segment... Dec 26 10:18:14.609112198 (498h49m14.480195243s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.609463647 (498h49m14.480546682s) [seg_pos={0 0}, upload=437816]: Done begin upload segment. Dec 26 10:18:14.609491941 (498h49m14.480574976s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610040396 (498h49m14.481123431s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610496985 (498h49m14.48158002s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610774714 (498h49m14.481857759s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611043024 (498h49m14.482126059s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611318639 (498h49m14.482401674s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611622909 (498h49m14.482705944s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.613638747 (498h49m14.484721782s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.614282773 (498h49m14.485365818s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.615846651 (498h49m14.486929696s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.617135205 (498h49m14.48821824s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.618487420 (498h49m14.489570465s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.619625988 (498h49m14.490709023s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.620824310 (498h49m14.491907345s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.622163480 (498h49m14.493246515s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:16.331618757 (498h49m16.202701802s) [upload=437816]: Next returned nil segment Dec 26 10:18:16.331635610 (498h49m16.202718645s) [upload=437816]: Waiting for error group managing segments... Dec 26 10:18:16.612650381 (498h49m16.483733416s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.622538299 (498h49m16.493621334s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.623481074 (498h49m16.494564119s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629239659 (498h49m16.500322704s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629900467 (498h49m16.500983522s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.632799878 (498h49m16.503882913s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.639602862 (498h49m16.510685897s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.644057315 (498h49m16.51514036s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.827488098 (498h49m16.698571143s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.933839800 (498h49m16.804922845s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.970575471 (498h49m16.841658526s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.046139899 (498h49m16.917222944s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615175711 (498h49m17.486258747s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615200027 (498h49m17.486283062s) [seg_pos={0 0}, upload=437816]: Segment reached optimal threshold of 13 pieces. Dec 26 10:18:17.615451246 (498h49m17.486534291s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615459532 (498h49m17.486542567s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615666807 (498h49m17.486749842s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615674000 (498h49m17.486757045s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615776666 (498h49m17.486859711s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615825538 (498h49m17.486908583s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615871687 (498h49m17.486954722s) [seg_pos={0 0}, upload=437816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:18:17.616073501 (498h49m17.487156556s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:17.616081616 (498h49m17.487164662s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:17.904101796 (498h49m17.775184841s) [upload=437816]: Done uploading segments... Dec 26 10:18:17.904125321 (498h49m17.775208356s) [upload=437816]: Upload finished. err: <nil> Duration: 6.009474099s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X......X.........X..................X.......XXX] duration: 1m0.002158617s trace: https://jaeger.dev.storj.io/trace/7aef545d37216976 FAIL (2025-12-26 10:19:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21:4883 , piece_id=45ESRA6YTW43MKEFIE2U3HPNW7JTPXM2SLNTHKY32F2YFORNVAQQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.441584147 (498h49m27.312667192s) [noise=true , address=64.49.13.137:4503 , piece_id=ZVP3DFOTW5GIW6WZ4KDQVV6GQV5MUS5VWLU5FDCAGFYFDDT2D5RA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.441837960 (498h49m27.312921005s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442146617 (498h49m27.313229653s) [noise=true , address=77.90.31.11:4560 , piece_id=G4EXQLB4I3GFTTLZNRTPTWRY2HUIBAMOTAAM333OJQB2SNZLQW7Q, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442525168 (498h49m27.313608214s) [noise=true , address=145.79.227.52:4458 , piece_id=COMKCWPTVIBRPOPEWIDYU73EFQFXWVQY7IUFEM42HRQWXGJOUZ7A, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442608046 (498h49m27.313691091s) [noise=true , address=77.90.31.15:4730 , piece_id=RMDDZY2KUS6A4SLQW2B6JTHANDBCHGRWJIRHQQJ4X635I53B3ZHQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442903218 (498h49m27.313986264s) [noise=true , address=145.79.227.34:4915 , piece_id=ADW24WPJ5YJ73F5ABFTK3QSON3TBWUMZNCWNNBMV6ODFWNRFGX5A, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.442912065 (498h49m27.3139951s) [noise=true , address=77.90.31.18:4837 , piece_id=OC6GNBCZQMJCV7JSF7XHO3YHE7HMZ3HBWBKBYJRDKLMKMCYN2UFQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443184915 (498h49m27.31426796s) [noise=true , address=77.90.31.14:4680 , piece_id=JK2NZRA5QXJJ2KWIQ3KPZT2GK2RJ3R6YGWVBLUHLGDYKBWUQKCIQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443385607 (498h49m27.314468652s) [noise=true , address=145.79.227.33:4876 , piece_id=OCV2FU5GXO5Q5TZ7CQ4WPGHSZ2MWFAFCP3VEX7TVGHELVNZJIGFA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443486720 (498h49m27.314569765s) [noise=true , address=45.91.116.12:4245 , piece_id=B5SSUXAQ7FF2WZKQGROIYQSZAMYR4M7K5WWA64O7KY46TEWWZQPQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.443777884 (498h49m27.314860919s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444156695 (498h49m27.31523973s) [noise=true , address=77.90.31.16:4762 , piece_id=TO4OWX4NFCLFXXPW7NQWJ7NVXWG24TTBXN4ISBZE667U27FPTXQA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444525026 (498h49m27.315608101s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:27.444959794 (498h49m27.31604284s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Uploading piece... Dec 26 10:18:28.898009100 (498h49m28.769092155s) [upload=437819]: Next returned nil segment Dec 26 10:18:28.898024930 (498h49m28.769107976s) [upload=437819]: Waiting for error group managing segments... Dec 26 10:18:29.317608319 (498h49m29.188691364s) [noise=true , address=145.79.227.52:4458 , piece_id=COMKCWPTVIBRPOPEWIDYU73EFQFXWVQY7IUFEM42HRQWXGJOUZ7A, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.417048480 (498h49m29.288131515s) [noise=true , address=64.49.13.137:4503 , piece_id=ZVP3DFOTW5GIW6WZ4KDQVV6GQV5MUS5VWLU5FDCAGFYFDDT2D5RA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.431472206 (498h49m29.302555241s) [noise=true , address=145.79.227.34:4915 , piece_id=ADW24WPJ5YJ73F5ABFTK3QSON3TBWUMZNCWNNBMV6ODFWNRFGX5A, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.531132256 (498h49m29.402215291s) [noise=true , address=77.90.31.14:4680 , piece_id=JK2NZRA5QXJJ2KWIQ3KPZT2GK2RJ3R6YGWVBLUHLGDYKBWUQKCIQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.539874212 (498h49m29.410957247s) [noise=true , address=45.91.116.12:4245 , piece_id=B5SSUXAQ7FF2WZKQGROIYQSZAMYR4M7K5WWA64O7KY46TEWWZQPQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.575613335 (498h49m29.44669638s) [noise=true , address=77.90.31.21:4883 , piece_id=45ESRA6YTW43MKEFIE2U3HPNW7JTPXM2SLNTHKY32F2YFORNVAQQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.588530461 (498h49m29.459613496s) [noise=true , address=77.90.31.17:4780 , piece_id=OXLKBKFLRXJFLRHIQ3ROAYBDCJUBZRIZ6U3RCZJOGK7Q7BXG26CQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.606622181 (498h49m29.477705216s) [noise=true , address=145.79.227.33:4876 , piece_id=OCV2FU5GXO5Q5TZ7CQ4WPGHSZ2MWFAFCP3VEX7TVGHELVNZJIGFA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.618087524 (498h49m29.489170559s) [noise=true , address=77.90.31.16:4762 , piece_id=TO4OWX4NFCLFXXPW7NQWJ7NVXWG24TTBXN4ISBZE667U27FPTXQA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.618804549 (498h49m29.489887594s) [noise=true , address=77.90.31.15:4730 , piece_id=RMDDZY2KUS6A4SLQW2B6JTHANDBCHGRWJIRHQQJ4X635I53B3ZHQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.627317289 (498h49m29.498400324s) [noise=true , address=77.90.31.18:4837 , piece_id=OC6GNBCZQMJCV7JSF7XHO3YHE7HMZ3HBWBKBYJRDKLMKMCYN2UFQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.743470775 (498h49m29.61455381s) [noise=true , address=77.90.31.19:4863 , piece_id=OQ7UPZNHXMG52A52UUIMOBDRI4B2EQE5VURHZM4PCOKYHVAG2C2Q, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:29.988678213 (498h49m29.859761248s) [noise=true , address=77.90.31.11:4560 , piece_id=G4EXQLB4I3GFTTLZNRTPTWRY2HUIBAMOTAAM333OJQB2SNZLQW7Q, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:30.173673986 (498h49m30.044757031s) [noise=true , address=145.79.227.44:4222 , piece_id=XU4TQPQXW4CXIJ3SI634Z32MMJB7GFQFHKNBZEGNHVONUQSVTSYQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: <nil> Dec 26 10:18:30.173691770 (498h49m30.044774815s) [seg_pos={0 0}, upload=437819]: Segment reached optimal threshold of 14 pieces. Dec 26 10:18:30.173872264 (498h49m30.044955299s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:30.173880039 (498h49m30.044963084s) [noise=true , address=45.91.116.10:4097 , piece_id=BP4CJA2CH3AHIR5VECJTPURNZ3T5KGO6J27SFTP3NLJJZBLPGWUQ, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174008883 (498h49m30.045091918s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49604->64.49.13.169:4706: use of closed network connection Dec 26 10:18:30.174017961 (498h49m30.045100996s) [noise=true , address=64.49.13.169:4706 , piece_id=34AJVQURKNGPGV7KDIVOSRIQRBK7THPE5V3WHLYCBMFHKJ5P3YLQ, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174049742 (498h49m30.045132777s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:30.174058829 (498h49m30.045141864s) [noise=true , address=145.79.227.30:4785 , piece_id=K4FBPYYIMVW7PKZL745ICIDGPR6BEBK3RU5IVHM5MHHCIWZFZEWA, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174254883 (498h49m30.045337918s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jEyiUUU9NCVFMU8frNggxTTYBMkehsqN4GLsFRkrxubdobGY2): protocol: write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection; write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47542->77.90.31.10:4527: use of closed network connection Dec 26 10:18:30.174261886 (498h49m30.045344921s) [noise=true , address=77.90.31.10:4527 , piece_id=D6A6HKGBC5M4IFNNJOG3VLJDM2JW6EELRH6YH2D5ZKPIRSLFMQQA, seg_pos={0 0}, upload=437819]: Long tail cancellation due to optimal threshold Dec 26 10:18:30.174347448 (498h49m30.045430493s) [seg_pos={0 0}, upload=437819]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:18:30.174415378 (498h49m30.045498413s) [upload=437819]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:30.174419666 (498h49m30.045502701s) [upload=437819]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:30.472374192 (498h49m30.343457227s) [upload=437819]: Done uploading segments... Dec 26 10:18:30.472393109 (498h49m30.343476144s) [upload=437819]: Upload finished. err: <nil> Duration: 1m0.002168325s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [X...X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.] duration: 6.059578814s trace: https://jaeger.dev.storj.io/trace/27e7e2f289849627 FAIL (2025-12-26 10:18:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ) context deadline exceeded ======LOGS====== Dec 26 10:18:14.364246752 (498h49m14.235329787s) [upload=437816]: Starting upload Dec 26 10:18:14.364320452 (498h49m14.235403498s) [upload=437816]: Done starting upload Dec 26 10:18:14.364425312 (498h49m14.235508347s) [upload=437816]: Uploading segments... Dec 26 10:18:14.364586949 (498h49m14.235669994s) [upload=437816]: Got next segment. Inline: false Dec 26 10:18:14.364639129 (498h49m14.235722174s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:18:14.364645762 (498h49m14.235728827s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:18:14.609052935 (498h49m14.48013597s) [seg_pos={0 0}, upload=437816]: Begin upload segment... Dec 26 10:18:14.609112198 (498h49m14.480195243s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.609463647 (498h49m14.480546682s) [seg_pos={0 0}, upload=437816]: Done begin upload segment. Dec 26 10:18:14.609491941 (498h49m14.480574976s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610040396 (498h49m14.481123431s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610496985 (498h49m14.48158002s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610774714 (498h49m14.481857759s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611043024 (498h49m14.482126059s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611318639 (498h49m14.482401674s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611622909 (498h49m14.482705944s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.613638747 (498h49m14.484721782s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.614282773 (498h49m14.485365818s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.615846651 (498h49m14.486929696s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.617135205 (498h49m14.48821824s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.618487420 (498h49m14.489570465s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.619625988 (498h49m14.490709023s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.620824310 (498h49m14.491907345s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.622163480 (498h49m14.493246515s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:16.331618757 (498h49m16.202701802s) [upload=437816]: Next returned nil segment Dec 26 10:18:16.331635610 (498h49m16.202718645s) [upload=437816]: Waiting for error group managing segments... Dec 26 10:18:16.612650381 (498h49m16.483733416s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.622538299 (498h49m16.493621334s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.623481074 (498h49m16.494564119s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629239659 (498h49m16.500322704s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629900467 (498h49m16.500983522s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.632799878 (498h49m16.503882913s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.639602862 (498h49m16.510685897s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.644057315 (498h49m16.51514036s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.827488098 (498h49m16.698571143s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.933839800 (498h49m16.804922845s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.970575471 (498h49m16.841658526s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.046139899 (498h49m16.917222944s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615175711 (498h49m17.486258747s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615200027 (498h49m17.486283062s) [seg_pos={0 0}, upload=437816]: Segment reached optimal threshold of 13 pieces. Dec 26 10:18:17.615451246 (498h49m17.486534291s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615459532 (498h49m17.486542567s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615666807 (498h49m17.486749842s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615674000 (498h49m17.486757045s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615776666 (498h49m17.486859711s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615825538 (498h49m17.486908583s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615871687 (498h49m17.486954722s) [seg_pos={0 0}, upload=437816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:18:17.616073501 (498h49m17.487156556s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:17.616081616 (498h49m17.487164662s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:17.904101796 (498h49m17.775184841s) [upload=437816]: Done uploading segments... Dec 26 10:18:17.904125321 (498h49m17.775208356s) [upload=437816]: Upload finished. err: <nil> Duration: 6.059604985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.X...X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX] duration: 30.002556488s trace: https://jaeger.dev.storj.io/trace/27e7e2f289849627 FAIL (2025-12-26 10:18:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ) context deadline exceeded ======LOGS====== Dec 26 10:18:14.364246752 (498h49m14.235329787s) [upload=437816]: Starting upload Dec 26 10:18:14.364320452 (498h49m14.235403498s) [upload=437816]: Done starting upload Dec 26 10:18:14.364425312 (498h49m14.235508347s) [upload=437816]: Uploading segments... Dec 26 10:18:14.364586949 (498h49m14.235669994s) [upload=437816]: Got next segment. Inline: false Dec 26 10:18:14.364639129 (498h49m14.235722174s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:18:14.364645762 (498h49m14.235728827s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:18:14.609052935 (498h49m14.48013597s) [seg_pos={0 0}, upload=437816]: Begin upload segment... Dec 26 10:18:14.609112198 (498h49m14.480195243s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.609463647 (498h49m14.480546682s) [seg_pos={0 0}, upload=437816]: Done begin upload segment. Dec 26 10:18:14.609491941 (498h49m14.480574976s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610040396 (498h49m14.481123431s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610496985 (498h49m14.48158002s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.610774714 (498h49m14.481857759s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611043024 (498h49m14.482126059s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611318639 (498h49m14.482401674s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.611622909 (498h49m14.482705944s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.613638747 (498h49m14.484721782s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.614282773 (498h49m14.485365818s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.615846651 (498h49m14.486929696s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.617135205 (498h49m14.48821824s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.618487420 (498h49m14.489570465s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.619625988 (498h49m14.490709023s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.620824310 (498h49m14.491907345s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:14.622163480 (498h49m14.493246515s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Uploading piece... Dec 26 10:18:16.331618757 (498h49m16.202701802s) [upload=437816]: Next returned nil segment Dec 26 10:18:16.331635610 (498h49m16.202718645s) [upload=437816]: Waiting for error group managing segments... Dec 26 10:18:16.612650381 (498h49m16.483733416s) [noise=true , address=145.79.227.11:4730 , piece_id=425M2RAMJF6ER5NHM34VUPGAU7QJMGFPPPC6EZCXGJHQ332OFGFQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.622538299 (498h49m16.493621334s) [noise=true , address=145.79.227.16:4910 , piece_id=AVHOHARJVKO3QNMJKEKIT45NQRUR62PAFE5W5K5VK7WHKVVO2LNA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.623481074 (498h49m16.494564119s) [noise=true , address=145.79.227.41:4087 , piece_id=TLXUTHELO2KHOKMWYIKBRYVVMDKQA62HNNMDMBZIASPBZJZQQB3A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629239659 (498h49m16.500322704s) [noise=true , address=145.79.227.36:4517 , piece_id=T5QLZL6KI25SPFTJ6JJO3M3A2M5ZL5Z3FWZ4WBK2CH6EIQTKV3GA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.629900467 (498h49m16.500983522s) [noise=true , address=145.79.227.35:4481 , piece_id=YDJYYZX2OIJAIAW5Y3RTHLP2MQVSY7C6OFU3MKPIJOR37C2KLYDQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.632799878 (498h49m16.503882913s) [noise=true , address=145.79.227.38:4979 , piece_id=DQZHP5YIBNQM6XHZKSNRKJ46DGBIR5THQGRRK6XN5HBDPT5D52ZA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.639602862 (498h49m16.510685897s) [noise=true , address=145.79.227.19:4078 , piece_id=HHC3JKVKI7MX2HBHVVUJFKFSPRUFVVC2F4LM5JUYQMN7PCW3TPPA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.644057315 (498h49m16.51514036s) [noise=true , address=145.79.227.39:4015 , piece_id=J5Q6BQY4EYZLCLGBVHBLZRVDT2XFLPYDKY2YR7HFEERZ6V556A2A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.827488098 (498h49m16.698571143s) [noise=true , address=45.91.116.8:4047 , piece_id=PVE4U2RK6DHO3L6JPOQVYAR3EFCTXFHRF2TW5I2SELHOZ4KTE2TQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.933839800 (498h49m16.804922845s) [noise=true , address=77.90.31.17:4770 , piece_id=J5FQWRL7I6SEAHVDTNKQ5LH3Z2CIY3B32ZIUPQVX5UQG5QCA6KCA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:16.970575471 (498h49m16.841658526s) [noise=true , address=45.91.116.11:4119 , piece_id=6BOCG7LGXW3QR2KVTRQR6GPZ3QRCG2NQAHMDR3DBKQU77DF5DBGQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.046139899 (498h49m16.917222944s) [noise=true , address=77.90.31.11:4556 , piece_id=6PTM5WD7FJOW2LTSFFDCFPUUZIPV3B3UTZNCP7CFD7V7X7ZNZUXA, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615175711 (498h49m17.486258747s) [noise=true , address=77.90.31.14:4664 , piece_id=UMUAIMG2BKNTSMOADQWSF4CGIKRNYEL4FC6ZANXHVSFHCRVTISUQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: <nil> Dec 26 10:18:17.615200027 (498h49m17.486283062s) [seg_pos={0 0}, upload=437816]: Segment reached optimal threshold of 13 pieces. Dec 26 10:18:17.615451246 (498h49m17.486534291s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615459532 (498h49m17.486542567s) [noise=true , address=45.91.116.12:4227 , piece_id=E343FEN2BUHNPXOZISCRQ2CWUBB5MPN4E5QTALPGMEXIKGY5N5QQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615666807 (498h49m17.486749842s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615674000 (498h49m17.486757045s) [noise=true , address=45.91.116.10:4081 , piece_id=LMXQCD3XRUKL26QRIH655QBCO3BPPYII6YIV3A7EMLP6TLVPYS6A, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615776666 (498h49m17.486859711s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 10:18:17.615825538 (498h49m17.486908583s) [noise=true , address=45.91.116.6:4973 , piece_id=X7IFQLWBVVT7KQTONH4SQM2QPEM3XFZMCMM543IYMZ65OA64VMMQ, seg_pos={0 0}, upload=437816]: Long tail cancellation due to optimal threshold Dec 26 10:18:17.615871687 (498h49m17.486954722s) [seg_pos={0 0}, upload=437816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:18:17.616073501 (498h49m17.487156556s) [upload=437816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:18:17.616081616 (498h49m17.487164662s) [upload=437816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:18:17.904101796 (498h49m17.775184841s) [upload=437816]: Done uploading segments... Dec 26 10:18:17.904125321 (498h49m17.775208356s) [upload=437816]: Upload finished. err: <nil> Duration: 30.002578679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X......X.........X..................X.......XX] duration: 1m0.002709636s trace: https://jaeger.dev.storj.io/trace/4b0e60d67a21f71 FAIL (2025-12-26 10:18:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.17:4002 , piece_id=HT64DQ4RXBHKGL6FSCU2H25LQC5FDDJTV7A2QXV5RD7OWZHGZHUQ, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.204494071 (498h48m27.075577106s) [noise=true , address=77.90.31.21:4885 , piece_id=7ESHEVBVXR2FWIOXMZAAJRC4EWFJQJ4IXLSUSXPOBVYUBGUY26AA, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.204929339 (498h48m27.076012374s) [noise=true , address=45.91.116.13:4167 , piece_id=OIWQESFVSE7DKDI6SGU536F36FS4WCDSYNYMLKENYIJ24VDS227Q, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.205327518 (498h48m27.076410553s) [noise=true , address=77.90.31.13:4644 , piece_id=HJ7UI6ONOD53YCGUK5QEUA7TZ56MCJDMG6MIV7KU6MDHI3ZD4OJA, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.205672204 (498h48m27.076755249s) [noise=true , address=145.79.227.44:4219 , piece_id=DRP4FMTB4ZZA7QOHRCH7R2P6DON466ZVUVKIJ3AUNK2XJWCYD3IQ, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.205886061 (498h48m27.076969097s) [noise=true , address=45.91.116.8:4056 , piece_id=6KHBJYXMJKA52F2K6YMBLB5AQL2CG32SL3GJUAHCMHX36B27THCQ, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.206529777 (498h48m27.077612812s) [noise=true , address=64.49.13.27:4965 , piece_id=UPHDAIJJAASWOSUPDGYLQRCXQPHC7RQYLJBBDVZ63FIATAY5KQ3Q, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.207182710 (498h48m27.078265745s) [noise=true , address=77.90.31.16:4759 , piece_id=JS7JEMQ2FRXN3O5LP524QKPTK7QFJKYZWIUVLUYOC3KKWAKVJM4Q, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.207283292 (498h48m27.078366327s) [noise=true , address=77.90.31.18:4827 , piece_id=UMY5MVUZF3IDP2LUSBY7AE2ALR7373QHA7G664XZ6XUCGCW6V6SA, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.207704444 (498h48m27.078787479s) [noise=true , address=45.91.116.5:4312 , piece_id=ZIPDIUWAYUZMLRVQYS7RV6FZU62TV5OQ4FGXUQXQN3HDPLR7LPEA, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.207743348 (498h48m27.078826383s) [noise=true , address=145.79.227.30:4795 , piece_id=SN7IK5I67EYP3DEGW3UWA4TDKZXQDGMGJJYUGVAK6UNGO6BPXYBQ, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.208107331 (498h48m27.079190366s) [noise=true , address=64.49.13.155:4924 , piece_id=ZJXHHXXABUXJOITGKWOLYF3KIAGFWWOJ2GOGFTBGWUGBXLRPXQEQ, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.208436949 (498h48m27.079519994s) [noise=true , address=145.79.227.52:4469 , piece_id=E66I2X3QYF7TXW6VGZS5EAMZ34WUVTUVVHEUUSUN33XTJHKZLZXA, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.208745837 (498h48m27.079828882s) [noise=true , address=77.90.31.14:4669 , piece_id=SEQBKNIHSXH2FDJO2XTANJJB2ECUAUM5TSMHAYXQ4SYE3NHR7RZQ, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.209231892 (498h48m27.080314937s) [noise=true , address=77.90.31.15:4727 , piece_id=7HSF42MR5BGMDWQXKOZ2O6IM34J52MQTWDNXL6O7LTXH4E4VIDIA, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:27.209640851 (498h48m27.080723886s) [noise=true , address=77.90.31.12:4601 , piece_id=CECWORR67B5C2X2NTB6M472RLJDPWME353HMRXPDHNQKJS6GNERQ, seg_pos={0 0}, upload=437807]: Uploading piece... Dec 26 10:17:28.552902062 (498h48m28.423985107s) [upload=437807]: Next returned nil segment Dec 26 10:17:28.552937029 (498h48m28.424020064s) [upload=437807]: Waiting for error group managing segments... Dec 26 10:17:28.898978431 (498h48m28.770061466s) [noise=true , address=64.49.13.155:4924 , piece_id=ZJXHHXXABUXJOITGKWOLYF3KIAGFWWOJ2GOGFTBGWUGBXLRPXQEQ, seg_pos={0 0}, upload=437807]: Done uploading piece. err: <nil> Dec 26 10:17:29.076496787 (498h48m28.947579832s) [noise=true , address=145.79.227.52:4469 , piece_id=E66I2X3QYF7TXW6VGZS5EAMZ34WUVTUVVHEUUSUN33XTJHKZLZXA, seg_pos={0 0}, upload=437807]: Done uploading piece. err: <nil> Dec 26 10:17:29.076975028 (498h48m28.948058073s) [noise=true , address=145.79.227.17:4002 , piece_id=HT64DQ4RXBHKGL6FSCU2H25LQC5FDDJTV7A2QXV5RD7OWZHGZHUQ, seg_pos={0 0}, upload=437807]: Done uploading piece. err: <nil> Dec 26 10:17:29.089702783 (498h48m28.960785818s) [noise=true , address=145.79.227.34:4936 , piece_id=AEGUGCMZMVGJAAXPEYXOKUOXEAX6GXSF6UBDBSIXXHFL6CCLWFZA, seg_pos={0 0}, upload=437807]: Done uploading piece. err: <nil> Dec 26 10:17:29.331965753 (498h48m29.203048798s) [noise=true , address=45.91.116.13:4167 , piece_id=OIWQESFVSE7DKDI6SGU536F36FS4WCDSYNYMLKENYIJ24VDS227Q, seg_pos={0 0}, upload=437807]: Done uploading piece. err: <nil> Dec 26 10:17:29.342488681 (498h48m29.213571726s) [noise=true , address=45.91.116.8:4056 , piece_id=6KHBJYXMJKA52F2K6YMBLB5AQL2CG32SL3GJUAHCMHX36B27THCQ, seg_pos={0 0}, upload=437807]: Done uploading piece. err: <nil> Dec 26 10:17:29.395872893 (498h48m29.266955928s) [noise=true , address=77.90.31.13:4644 , piece_id=HJ7UI6ONOD53YCGUK5QEUA7TZ56MCJDMG6MIV7KU6MDHI3ZD4OJA, seg_pos={0 0}, upload=437807]: Done uploading piece. err: <nil> Dec 26 10:17:29.405974068 (498h48m29.277057104s) [noise=true , address=77.90.31.19:4847 , piece_id=6EF764IQ3NXZ6BGFQJC5HM6H75RSECY4CWHSL3Z6XP2SDPH7XJMQ, seg_pos={0 0}, upload=437807]: Done uploading piece. err: <nil> Dec 26 10:17:29.409263171 (498h48m29.280346206s) [noise=true , address=77.90.31.12:4601 , piece_id=CECWORR67B5C2X2NTB6M472RLJDPWME353HMRXPDHNQKJS6GNERQ, seg_pos={0 0}, upload=437807]: Done uploading piece. err: <nil> Dec 26 10:17:29.429854972 (498h48m29.300938017s) [noise=true , address=77.90.31.14:4669 , piece_id=SEQBKNIHSXH2FDJO2XTANJJB2ECUAUM5TSMHAYXQ4SYE3NHR7RZQ, seg_pos={0 0}, upload=437807]: Done uploading piece. err: <nil> Dec 26 10:17:29.493114699 (498h48m29.364197734s) [noise=true , address=77.90.31.21:4885 , piece_id=7ESHEVBVXR2FWIOXMZAAJRC4EWFJQJ4IXLSUSXPOBVYUBGUY26AA, seg_pos={0 0}, upload=437807]: Done uploading piece. err: <nil> Dec 26 10:17:29.914955997 (498h48m29.786039032s) [noise=true , address=64.49.13.27:4965 , piece_id=UPHDAIJJAASWOSUPDGYLQRCXQPHC7RQYLJBBDVZ63FIATAY5KQ3Q, seg_pos={0 0}, upload=437807]: Done uploading piece. err: <nil> Dec 26 10:17:29.993758480 (498h48m29.864841515s) [noise=true , address=145.79.227.44:4219 , piece_id=DRP4FMTB4ZZA7QOHRCH7R2P6DON466ZVUVKIJ3AUNK2XJWCYD3IQ, seg_pos={0 0}, upload=437807]: Done uploading piece. err: <nil> Dec 26 10:17:30.005148740 (498h48m29.876231785s) [noise=true , address=145.79.227.30:4795 , piece_id=SN7IK5I67EYP3DEGW3UWA4TDKZXQDGMGJJYUGVAK6UNGO6BPXYBQ, seg_pos={0 0}, upload=437807]: Done uploading piece. err: <nil> Dec 26 10:17:30.005172194 (498h48m29.876255239s) [seg_pos={0 0}, upload=437807]: Segment reached optimal threshold of 14 pieces. Dec 26 10:17:30.005369751 (498h48m29.876452796s) [noise=true , address=77.90.31.15:4727 , piece_id=7HSF42MR5BGMDWQXKOZ2O6IM34J52MQTWDNXL6O7LTXH4E4VIDIA, seg_pos={0 0}, upload=437807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M): protocol: write tcp 188.166.225.170:49912->77.90.31.15:4727: use of closed network connection; write tcp 188.166.225.170:49912->77.90.31.15:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49912->77.90.31.15:4727: use of closed network connection Dec 26 10:17:30.005376042 (498h48m29.876459077s) [noise=true , address=77.90.31.15:4727 , piece_id=7HSF42MR5BGMDWQXKOZ2O6IM34J52MQTWDNXL6O7LTXH4E4VIDIA, seg_pos={0 0}, upload=437807]: Long tail cancellation due to optimal threshold Dec 26 10:17:30.005412321 (498h48m29.876495357s) [noise=true , address=77.90.31.18:4827 , piece_id=UMY5MVUZF3IDP2LUSBY7AE2ALR7373QHA7G664XZ6XUCGCW6V6SA, seg_pos={0 0}, upload=437807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 26 10:17:30.005420677 (498h48m29.876503712s) [noise=true , address=77.90.31.18:4827 , piece_id=UMY5MVUZF3IDP2LUSBY7AE2ALR7373QHA7G664XZ6XUCGCW6V6SA, seg_pos={0 0}, upload=437807]: Long tail cancellation due to optimal threshold Dec 26 10:17:30.005570002 (498h48m29.876653037s) [noise=true , address=77.90.31.16:4759 , piece_id=JS7JEMQ2FRXN3O5LP524QKPTK7QFJKYZWIUVLUYOC3KKWAKVJM4Q, seg_pos={0 0}, upload=437807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 26 10:17:30.005577476 (498h48m29.876660511s) [noise=true , address=77.90.31.16:4759 , piece_id=JS7JEMQ2FRXN3O5LP524QKPTK7QFJKYZWIUVLUYOC3KKWAKVJM4Q, seg_pos={0 0}, upload=437807]: Long tail cancellation due to optimal threshold Dec 26 10:17:30.005593556 (498h48m29.876676591s) [noise=true , address=45.91.116.5:4312 , piece_id=ZIPDIUWAYUZMLRVQYS7RV6FZU62TV5OQ4FGXUQXQN3HDPLR7LPEA, seg_pos={0 0}, upload=437807]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gG7V9H3GkvqC8QmSqmitZaV6xSqR41Rgposb1xnDk8QkXEhFg): protocol: expected piece hash; context canceled; EOF Dec 26 10:17:30.005600581 (498h48m29.876683626s) [noise=true , address=45.91.116.5:4312 , piece_id=ZIPDIUWAYUZMLRVQYS7RV6FZU62TV5OQ4FGXUQXQN3HDPLR7LPEA, seg_pos={0 0}, upload=437807]: Long tail cancellation due to optimal threshold Dec 26 10:17:30.005620247 (498h48m29.876703292s) [seg_pos={0 0}, upload=437807]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:17:30.005693878 (498h48m29.876776913s) [upload=437807]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:17:30.005699127 (498h48m29.876782172s) [upload=437807]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:17:30.294667965 (498h48m30.165751s) [upload=437807]: Done uploading segments... Dec 26 10:17:30.294686469 (498h48m30.165769505s) [upload=437807]: Upload finished. err: <nil> Duration: 1m0.002733481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [X.X...X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....X] duration: 30.001451715s trace: https://jaeger.dev.storj.io/trace/6e4ba216573ced5d FAIL (2025-12-26 10:17:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 3XAN6EZUJXHVCDWXQQ5NKSTZNJIPI6UYWWT4OHMHVBVXOITPIN7Q) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: S2MRN2OS73UUCR2EFY4WOT75EIVHQ6UU6WMYTVPGUJFO633UBU3Q) context deadline exceeded ======LOGS====== 8h48m6.806645535s) [upload=437802]: Uploading segments... Dec 26 10:17:06.935728847 (498h48m6.806811892s) [upload=437802]: Got next segment. Inline: false Dec 26 10:17:06.935774804 (498h48m6.806857849s) [upload=437802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:17:06.935780606 (498h48m6.806863651s) [upload=437802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:17:07.180466180 (498h48m7.051549225s) [seg_pos={0 0}, upload=437802]: Begin upload segment... Dec 26 10:17:07.180541443 (498h48m7.051624488s) [seg_pos={0 0}, upload=437802]: Done begin upload segment. Dec 26 10:17:07.180594503 (498h48m7.051677538s) [noise=true , address=145.79.227.45:4229 , piece_id=YX6HCOFZSVRMES2XVXZDJQSST47KFPSFAFPMZGH2SMYCVKXKCXDA, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.180615814 (498h48m7.051698849s) [noise=true , address=145.79.227.16:4909 , piece_id=Q36NPVBUZDNLWOXQOFLIVOG6ZZRODP3RUKCA2XIAZZJU4BLXGEJQ, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.181169378 (498h48m7.052252413s) [noise=true , address=45.91.116.6:4973 , piece_id=RIBVY3GZ6NT7N345D2RRLOO4EO53ISGHXB25KLS6T3WCR6LJIFHQ, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.181460683 (498h48m7.052543728s) [noise=true , address=145.79.227.48:4302 , piece_id=KAVEQDKLRKZMCTBLKQCWNIEWC6QQAHX5AQQWRSMWD4DO23UMLUZQ, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.181777887 (498h48m7.052860922s) [noise=true , address=45.91.116.5:4300 , piece_id=PRI3LK7RW765ZEP3PQFEVARDLC6LMGRBNAJX3G2KNZR4HEL3ZQTQ, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.182188869 (498h48m7.053271914s) [noise=true , address=45.91.116.11:4120 , piece_id=3XAN6EZUJXHVCDWXQQ5NKSTZNJIPI6UYWWT4OHMHVBVXOITPIN7Q, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.182612957 (498h48m7.053695992s) [noise=true , address=45.91.116.12:4225 , piece_id=XGHXZD3N5WOHM25WEYN7M2IPJCZO2BYFQYXRSKKIFGJ63NIKN47A, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.182982030 (498h48m7.054065065s) [noise=true , address=77.90.31.19:4844 , piece_id=NIIUG776WWGKAEO6XZAM344IKSC6XY2SAS53H3LRD6VKFSWLRQ4Q, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.183423851 (498h48m7.054506887s) [noise=true , address=145.79.227.36:4517 , piece_id=WSP3VWYAWASXQDC3WYMVPBRKRO25U25SPJNFULYDPAQ6YMNMXB7Q, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.183786602 (498h48m7.054869637s) [noise=true , address=145.79.227.35:4481 , piece_id=7TB3QRCEEN5H77LOEBNFQ6LZ72NRZ55ABNT5DWKLKLQJJFIGOQDQ, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.184102533 (498h48m7.055185568s) [noise=true , address=145.79.227.24:4583 , piece_id=LFL74EATCYBPGYQGLQAZLUBYTK7L3V2OEVMFV73JQ3BQUO5YFQJA, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.184403928 (498h48m7.055486963s) [noise=true , address=145.79.227.5:4222 , piece_id=UFMT7QXGHUYPSXK2FDHUFUKTVY47UXWLVOB4UTFBWSLQOMX347DA, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.185983655 (498h48m7.0570667s) [noise=true , address=145.79.227.30:4761 , piece_id=XS4IO2YZNZCIPPRYPJQXLVDBMNVSFLFYHFZSGJ2V3IZRKFVOMEKQ, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.186717623 (498h48m7.057800668s) [noise=true , address=145.79.227.3:4150 , piece_id=YAS7ZF4U2I7KS7YWSVRW32GVC4I3TNUSGXBAJD3JKPHTPCCYW4YQ, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.187155908 (498h48m7.058238953s) [noise=true , address=45.91.116.8:4047 , piece_id=S2MRN2OS73UUCR2EFY4WOT75EIVHQ6UU6WMYTVPGUJFO633UBU3Q, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:07.187618388 (498h48m7.058701463s) [noise=true , address=145.79.227.42:4121 , piece_id=XK6RHL43SN7Z6BVPZNSHVGHC6PMP77Y6FILJNOLQC7FM6T6QCCWA, seg_pos={0 0}, upload=437802]: Uploading piece... Dec 26 10:17:08.807653474 (498h48m8.678736509s) [upload=437802]: Next returned nil segment Dec 26 10:17:08.807677370 (498h48m8.678760405s) [upload=437802]: Waiting for error group managing segments... Dec 26 10:17:09.194138820 (498h48m9.065221865s) [noise=true , address=145.79.227.36:4517 , piece_id=WSP3VWYAWASXQDC3WYMVPBRKRO25U25SPJNFULYDPAQ6YMNMXB7Q, seg_pos={0 0}, upload=437802]: Done uploading piece. err: <nil> Dec 26 10:17:09.197573439 (498h48m9.068656474s) [noise=true , address=145.79.227.16:4909 , piece_id=Q36NPVBUZDNLWOXQOFLIVOG6ZZRODP3RUKCA2XIAZZJU4BLXGEJQ, seg_pos={0 0}, upload=437802]: Done uploading piece. err: <nil> Dec 26 10:17:09.197872339 (498h48m9.068955384s) [noise=true , address=145.79.227.5:4222 , piece_id=UFMT7QXGHUYPSXK2FDHUFUKTVY47UXWLVOB4UTFBWSLQOMX347DA, seg_pos={0 0}, upload=437802]: Done uploading piece. err: <nil> Dec 26 10:17:09.199379188 (498h48m9.070462223s) [noise=true , address=145.79.227.48:4302 , piece_id=KAVEQDKLRKZMCTBLKQCWNIEWC6QQAHX5AQQWRSMWD4DO23UMLUZQ, seg_pos={0 0}, upload=437802]: Done uploading piece. err: <nil> Dec 26 10:17:09.200382037 (498h48m9.071465072s) [noise=true , address=145.79.227.35:4481 , piece_id=7TB3QRCEEN5H77LOEBNFQ6LZ72NRZ55ABNT5DWKLKLQJJFIGOQDQ, seg_pos={0 0}, upload=437802]: Done uploading piece. err: <nil> Dec 26 10:17:09.201587262 (498h48m9.072670307s) [noise=true , address=145.79.227.3:4150 , piece_id=YAS7ZF4U2I7KS7YWSVRW32GVC4I3TNUSGXBAJD3JKPHTPCCYW4YQ, seg_pos={0 0}, upload=437802]: Done uploading piece. err: <nil> Dec 26 10:17:09.203150769 (498h48m9.074233804s) [noise=true , address=145.79.227.45:4229 , piece_id=YX6HCOFZSVRMES2XVXZDJQSST47KFPSFAFPMZGH2SMYCVKXKCXDA, seg_pos={0 0}, upload=437802]: Done uploading piece. err: <nil> Dec 26 10:17:09.216695191 (498h48m9.087778226s) [noise=true , address=145.79.227.42:4121 , piece_id=XK6RHL43SN7Z6BVPZNSHVGHC6PMP77Y6FILJNOLQC7FM6T6QCCWA, seg_pos={0 0}, upload=437802]: Done uploading piece. err: <nil> Dec 26 10:17:09.221716211 (498h48m9.092799247s) [noise=true , address=145.79.227.24:4583 , piece_id=LFL74EATCYBPGYQGLQAZLUBYTK7L3V2OEVMFV73JQ3BQUO5YFQJA, seg_pos={0 0}, upload=437802]: Done uploading piece. err: <nil> Dec 26 10:17:09.426675288 (498h48m9.297758323s) [noise=true , address=45.91.116.11:4120 , piece_id=3XAN6EZUJXHVCDWXQQ5NKSTZNJIPI6UYWWT4OHMHVBVXOITPIN7Q, seg_pos={0 0}, upload=437802]: Done uploading piece. err: <nil> Dec 26 10:17:09.525762498 (498h48m9.396845533s) [noise=true , address=45.91.116.6:4973 , piece_id=RIBVY3GZ6NT7N345D2RRLOO4EO53ISGHXB25KLS6T3WCR6LJIFHQ, seg_pos={0 0}, upload=437802]: Done uploading piece. err: <nil> Dec 26 10:17:09.541036492 (498h48m9.412119537s) [noise=true , address=45.91.116.12:4225 , piece_id=XGHXZD3N5WOHM25WEYN7M2IPJCZO2BYFQYXRSKKIFGJ63NIKN47A, seg_pos={0 0}, upload=437802]: Done uploading piece. err: <nil> Dec 26 10:17:09.569724038 (498h48m9.440807073s) [noise=true , address=45.91.116.8:4047 , piece_id=S2MRN2OS73UUCR2EFY4WOT75EIVHQ6UU6WMYTVPGUJFO633UBU3Q, seg_pos={0 0}, upload=437802]: Done uploading piece. err: <nil> Dec 26 10:17:09.569744257 (498h48m9.440827292s) [seg_pos={0 0}, upload=437802]: Segment reached optimal threshold of 13 pieces. Dec 26 10:17:09.569976088 (498h48m9.441059123s) [noise=true , address=145.79.227.30:4761 , piece_id=XS4IO2YZNZCIPPRYPJQXLVDBMNVSFLFYHFZSGJ2V3IZRKFVOMEKQ, seg_pos={0 0}, upload=437802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 10:17:09.569982450 (498h48m9.441065495s) [noise=true , address=145.79.227.30:4761 , piece_id=XS4IO2YZNZCIPPRYPJQXLVDBMNVSFLFYHFZSGJ2V3IZRKFVOMEKQ, seg_pos={0 0}, upload=437802]: Long tail cancellation due to optimal threshold Dec 26 10:17:09.570391158 (498h48m9.441474204s) [noise=true , address=77.90.31.19:4844 , piece_id=NIIUG776WWGKAEO6XZAM344IKSC6XY2SAS53H3LRD6VKFSWLRQ4Q, seg_pos={0 0}, upload=437802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:46916->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:46916->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46916->77.90.31.19:4844: use of closed network connection Dec 26 10:17:09.570400476 (498h48m9.441483521s) [noise=true , address=77.90.31.19:4844 , piece_id=NIIUG776WWGKAEO6XZAM344IKSC6XY2SAS53H3LRD6VKFSWLRQ4Q, seg_pos={0 0}, upload=437802]: Long tail cancellation due to optimal threshold Dec 26 10:17:09.570411297 (498h48m9.441494342s) [noise=true , address=45.91.116.5:4300 , piece_id=PRI3LK7RW765ZEP3PQFEVARDLC6LMGRBNAJX3G2KNZR4HEL3ZQTQ, seg_pos={0 0}, upload=437802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:36170->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:36170->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36170->45.91.116.5:4300: use of closed network connection Dec 26 10:17:09.570419703 (498h48m9.441502738s) [noise=true , address=45.91.116.5:4300 , piece_id=PRI3LK7RW765ZEP3PQFEVARDLC6LMGRBNAJX3G2KNZR4HEL3ZQTQ, seg_pos={0 0}, upload=437802]: Long tail cancellation due to optimal threshold Dec 26 10:17:09.570437937 (498h48m9.441520982s) [seg_pos={0 0}, upload=437802]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:17:09.570512340 (498h48m9.441595375s) [upload=437802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:17:09.570517630 (498h48m9.441600665s) [upload=437802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:17:09.863370622 (498h48m9.734453667s) [upload=437802]: Done uploading segments... Dec 26 10:17:09.863396020 (498h48m9.734479055s) [upload=437802]: Upload finished. err: <nil> Duration: 30.001473457s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X......X.........X..................X.......X] duration: 1m0.00265702s trace: https://jaeger.dev.storj.io/trace/165837d63c75f632 FAIL (2025-12-26 10:17:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=77.90.31.11:4577 , piece_id=ROKGBWOSXC2KTJD5ZM3TTYEQEKRJWZZCIGHYIB2PKNVYFFAGX5OA, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.628181908 (498h47m21.499264953s) [noise=true , address=77.90.31.19:4866 , piece_id=HJKRTL5WBWXNTBWQC3NECBH2JWA7IKSALAOTYMAGLU2CIEGQHBDA, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.628660590 (498h47m21.499743635s) [noise=true , address=77.90.31.14:4671 , piece_id=SWWKLPZRL7LMLJU2IHJSBO7SMAPQNLSUOITWJTZM4BZLHZYCC5LA, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.628910466 (498h47m21.499993501s) [noise=true , address=145.79.227.44:4196 , piece_id=FJHDQVI5BIBHGY7SDZ3NAIH4VKJGOIO73WZHKNA4UFPUIHC3KECQ, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.629174128 (498h47m21.500257173s) [noise=true , address=145.79.227.34:4930 , piece_id=32Q2BBDSMCSHE6ROJFBOFWGRAVJMHTXRNRMBL5N5RJNQELGVEURA, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.629380159 (498h47m21.500463195s) [noise=true , address=77.90.31.12:4608 , piece_id=OX7PJB6DZRZVHXHOIAR6F7RUPEYHICIY2HSMU3CHNXZ2MXNMMH3Q, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.629648090 (498h47m21.500731135s) [noise=true , address=145.79.227.33:4897 , piece_id=OD2YSQUZUGTTQLYIQ6OUHNYRHXWV3HAGXKZPRQAOECRUHMCQJDBQ, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.629769622 (498h47m21.500852657s) [noise=true , address=45.91.116.11:4145 , piece_id=2RRSNNDO3RGGHG365QR4EPADWK2NJUJZUTP3UJXR7YIL3ASOLU6A, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.630090503 (498h47m21.501173548s) [noise=true , address=64.49.13.155:4923 , piece_id=G5N6APFVYL2VITR7HHK3CNZABEUSZT45225I734O3KAVI2FTX5PA, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.630161718 (498h47m21.501244753s) [noise=true , address=145.79.227.20:4445 , piece_id=64ELV3DWPOPOQTIS4TMBZ2FPMD5DPNH6VBBGAWIGVGO2PDJ6IKMA, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.630673132 (498h47m21.501756177s) [noise=true , address=145.79.227.30:4772 , piece_id=65KBMLVPG2XV2Q45PPAH4T2WB4KGUIGSCKI4V2AIR2OYD23HXVJA, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.630748646 (498h47m21.501831691s) [noise=true , address=64.49.13.140:4516 , piece_id=AV6H6MIZBNQFUOCMU6MCMXGOWXIBT2RCHD3N753IERRT22VJ47HQ, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.631124051 (498h47m21.502207096s) [noise=true , address=77.90.31.13:4650 , piece_id=36JGLZ3B766BQ5F5DU7Y35P235TOHAKVQ6ANAKOLFJ3J2H6QP5SA, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.631156582 (498h47m21.502239627s) [noise=true , address=45.91.116.12:4245 , piece_id=UFOR5CAJSYBEZ4IAT6JBB4S455VFV5MGLBSQCRAUJNAAG65GEEJA, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.635549638 (498h47m21.506632673s) [noise=true , address=77.90.31.16:4759 , piece_id=MVQSGOV2EHM6WAW3NH4GLWIUKINRJL4SO4OPJ2JPFOQRRJ565UQA, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:21.636259149 (498h47m21.507342194s) [noise=true , address=145.79.227.40:4070 , piece_id=H56BDBAPQJKWHNUXSWVYR33YV2TJK3IF6MWBR5KAAVQJBIO32RVQ, seg_pos={0 0}, upload=437790]: Uploading piece... Dec 26 10:16:23.010982248 (498h47m22.882065293s) [upload=437790]: Next returned nil segment Dec 26 10:16:23.010999161 (498h47m22.882082206s) [upload=437790]: Waiting for error group managing segments... Dec 26 10:16:23.371436716 (498h47m23.242519761s) [noise=true , address=64.49.13.140:4516 , piece_id=AV6H6MIZBNQFUOCMU6MCMXGOWXIBT2RCHD3N753IERRT22VJ47HQ, seg_pos={0 0}, upload=437790]: Done uploading piece. err: <nil> Dec 26 10:16:23.523258321 (498h47m23.394341356s) [noise=true , address=145.79.227.20:4445 , piece_id=64ELV3DWPOPOQTIS4TMBZ2FPMD5DPNH6VBBGAWIGVGO2PDJ6IKMA, seg_pos={0 0}, upload=437790]: Done uploading piece. err: <nil> Dec 26 10:16:23.687936086 (498h47m23.559019131s) [noise=true , address=145.79.227.34:4930 , piece_id=32Q2BBDSMCSHE6ROJFBOFWGRAVJMHTXRNRMBL5N5RJNQELGVEURA, seg_pos={0 0}, upload=437790]: Done uploading piece. err: <nil> Dec 26 10:16:23.794046328 (498h47m23.665129373s) [noise=true , address=77.90.31.15:4723 , piece_id=WJALFIHYCX3ZRJ4H4KRGX36TDQ4ZOAAO7IC6FRY4AI6LP2GNSXUA, seg_pos={0 0}, upload=437790]: Done uploading piece. err: <nil> Dec 26 10:16:23.833095384 (498h47m23.704178429s) [noise=true , address=77.90.31.19:4866 , piece_id=HJKRTL5WBWXNTBWQC3NECBH2JWA7IKSALAOTYMAGLU2CIEGQHBDA, seg_pos={0 0}, upload=437790]: Done uploading piece. err: <nil> Dec 26 10:16:23.843279837 (498h47m23.714362883s) [noise=true , address=77.90.31.12:4608 , piece_id=OX7PJB6DZRZVHXHOIAR6F7RUPEYHICIY2HSMU3CHNXZ2MXNMMH3Q, seg_pos={0 0}, upload=437790]: Done uploading piece. err: <nil> Dec 26 10:16:23.997739104 (498h47m23.868822139s) [noise=true , address=45.91.116.11:4145 , piece_id=2RRSNNDO3RGGHG365QR4EPADWK2NJUJZUTP3UJXR7YIL3ASOLU6A, seg_pos={0 0}, upload=437790]: Done uploading piece. err: <nil> Dec 26 10:16:24.062876748 (498h47m23.933959793s) [noise=true , address=145.79.227.33:4897 , piece_id=OD2YSQUZUGTTQLYIQ6OUHNYRHXWV3HAGXKZPRQAOECRUHMCQJDBQ, seg_pos={0 0}, upload=437790]: Done uploading piece. err: <nil> Dec 26 10:16:24.101781098 (498h47m23.972864133s) [noise=true , address=77.90.31.18:4833 , piece_id=NZWWT7CAMSNOVGJGTWN4AQDU7N7KMTJ4NHTKKJY3IMKAV5O5QTWQ, seg_pos={0 0}, upload=437790]: Done uploading piece. err: <nil> Dec 26 10:16:24.103892098 (498h47m23.974975143s) [noise=true , address=77.90.31.16:4759 , piece_id=MVQSGOV2EHM6WAW3NH4GLWIUKINRJL4SO4OPJ2JPFOQRRJ565UQA, seg_pos={0 0}, upload=437790]: Done uploading piece. err: <nil> Dec 26 10:16:24.508828828 (498h47m24.379911863s) [noise=true , address=145.79.227.40:4070 , piece_id=H56BDBAPQJKWHNUXSWVYR33YV2TJK3IF6MWBR5KAAVQJBIO32RVQ, seg_pos={0 0}, upload=437790]: Done uploading piece. err: <nil> Dec 26 10:16:24.532019486 (498h47m24.403102531s) [noise=true , address=45.91.116.12:4245 , piece_id=UFOR5CAJSYBEZ4IAT6JBB4S455VFV5MGLBSQCRAUJNAAG65GEEJA, seg_pos={0 0}, upload=437790]: Done uploading piece. err: <nil> Dec 26 10:16:24.629716648 (498h47m24.500799683s) [noise=true , address=145.79.227.44:4196 , piece_id=FJHDQVI5BIBHGY7SDZ3NAIH4VKJGOIO73WZHKNA4UFPUIHC3KECQ, seg_pos={0 0}, upload=437790]: Done uploading piece. err: <nil> Dec 26 10:16:24.820988072 (498h47m24.692071107s) [noise=true , address=145.79.227.30:4772 , piece_id=65KBMLVPG2XV2Q45PPAH4T2WB4KGUIGSCKI4V2AIR2OYD23HXVJA, seg_pos={0 0}, upload=437790]: Done uploading piece. err: <nil> Dec 26 10:16:24.821007769 (498h47m24.692090804s) [seg_pos={0 0}, upload=437790]: Segment reached optimal threshold of 14 pieces. Dec 26 10:16:24.821157705 (498h47m24.692240741s) [noise=true , address=64.49.13.155:4923 , piece_id=G5N6APFVYL2VITR7HHK3CNZABEUSZT45225I734O3KAVI2FTX5PA, seg_pos={0 0}, upload=437790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: write tcp 188.166.225.170:34778->64.49.13.155:4923: use of closed network connection; write tcp 188.166.225.170:34778->64.49.13.155:4923: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34778->64.49.13.155:4923: use of closed network connection Dec 26 10:16:24.821163326 (498h47m24.692246361s) [noise=true , address=64.49.13.155:4923 , piece_id=G5N6APFVYL2VITR7HHK3CNZABEUSZT45225I734O3KAVI2FTX5PA, seg_pos={0 0}, upload=437790]: Long tail cancellation due to optimal threshold Dec 26 10:16:24.821211828 (498h47m24.692294863s) [noise=true , address=77.90.31.11:4577 , piece_id=ROKGBWOSXC2KTJD5ZM3TTYEQEKRJWZZCIGHYIB2PKNVYFFAGX5OA, seg_pos={0 0}, upload=437790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei): protocol: expected piece hash; context canceled; EOF Dec 26 10:16:24.821221115 (498h47m24.69230416s) [noise=true , address=77.90.31.11:4577 , piece_id=ROKGBWOSXC2KTJD5ZM3TTYEQEKRJWZZCIGHYIB2PKNVYFFAGX5OA, seg_pos={0 0}, upload=437790]: Long tail cancellation due to optimal threshold Dec 26 10:16:24.821293414 (498h47m24.692376459s) [noise=true , address=77.90.31.14:4671 , piece_id=SWWKLPZRL7LMLJU2IHJSBO7SMAPQNLSUOITWJTZM4BZLHZYCC5LA, seg_pos={0 0}, upload=437790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W): protocol: expected piece hash; context canceled; EOF Dec 26 10:16:24.821298904 (498h47m24.692381949s) [noise=true , address=77.90.31.14:4671 , piece_id=SWWKLPZRL7LMLJU2IHJSBO7SMAPQNLSUOITWJTZM4BZLHZYCC5LA, seg_pos={0 0}, upload=437790]: Long tail cancellation due to optimal threshold Dec 26 10:16:24.821422219 (498h47m24.692505254s) [noise=true , address=77.90.31.13:4650 , piece_id=36JGLZ3B766BQ5F5DU7Y35P235TOHAKVQ6ANAKOLFJ3J2H6QP5SA, seg_pos={0 0}, upload=437790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FNXVNAv3KeiRqTnsvQkMobyRcNFmPAAjVqSNvgs6MPJH4smVU): protocol: expected piece hash; context canceled; EOF Dec 26 10:16:24.821428230 (498h47m24.692511275s) [noise=true , address=77.90.31.13:4650 , piece_id=36JGLZ3B766BQ5F5DU7Y35P235TOHAKVQ6ANAKOLFJ3J2H6QP5SA, seg_pos={0 0}, upload=437790]: Long tail cancellation due to optimal threshold Dec 26 10:16:24.821508704 (498h47m24.692591749s) [seg_pos={0 0}, upload=437790]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:16:24.821596771 (498h47m24.692679806s) [upload=437790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:16:24.821601880 (498h47m24.692684916s) [upload=437790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:16:25.108071548 (498h47m24.979154593s) [upload=437790]: Done uploading segments... Dec 26 10:16:25.108099922 (498h47m24.979182967s) [upload=437790]: Upload finished. err: <nil> Duration: 1m0.002670448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X......X.........X..................X.......] duration: 33.405644355s trace: https://jaeger.dev.storj.io/trace/3eea9e0464bed0e7 FAIL (2025-12-26 10:16:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.333867109 (498h40m58.204950154s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.334611487 (498h40m58.205694542s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.335747089 (498h40m58.206830124s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336515553 (498h40m58.207598598s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336870569 (498h40m58.207953614s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337310336 (498h40m58.208393381s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337723112 (498h40m58.208806157s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338318425 (498h40m58.20940147s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338828727 (498h40m58.209911772s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339185056 (498h40m58.210268091s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339640863 (498h40m58.210723909s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339998325 (498h40m58.21108136s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.340445936 (498h40m58.211528981s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.341374635 (498h40m58.21245768s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:59.798768304 (498h40m59.669851359s) [upload=437697]: Next returned nil segment Dec 26 10:09:59.798783604 (498h40m59.669866649s) [upload=437697]: Waiting for error group managing segments... Dec 26 10:10:00.202844345 (498h41m0.07392739s) [noise=true , address=145.79.227.43:4175 , piece_id=5AW772QNP72OEKJTB6I4JGD4L5ONMRF5ZNXYLMKZAQDC56VOLSLQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.212993169 (498h41m0.084076204s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.224811985 (498h41m0.09589503s) [noise=true , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.239828058 (498h41m0.110911093s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.411596484 (498h41m0.282679529s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.490073348 (498h41m0.361156383s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.496009011 (498h41m0.367092056s) [noise=true , address=77.90.31.21:4904 , piece_id=NUBWDV2G25QBQKJDLQWVQG5JIE7PUW5XRHYNRCTVM3EENZPNQ6YA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.505536983 (498h41m0.376620018s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.621872607 (498h41m0.492955642s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.628231495 (498h41m0.49931453s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.634412284 (498h41m0.505495329s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.069090817 (498h41m0.940173852s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178406242 (498h41m1.049489277s) [noise=true , address=145.79.227.44:4226 , piece_id=CWN7Z7L2QIYNO75WNLF6Q4PPM3EIEAYJGYJCCY47U6NT4E4PYEVQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178763302 (498h41m1.049846347s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178785645 (498h41m1.04986869s) [seg_pos={0 0}, upload=437697]: Segment reached optimal threshold of 14 pieces. Dec 26 10:10:01.179093852 (498h41m1.050176897s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection Dec 26 10:10:01.179102047 (498h41m1.050185092s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179217547 (498h41m1.050300592s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection Dec 26 10:10:01.179223299 (498h41m1.050306334s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179319030 (498h41m1.050402075s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AZxy9wnMgVu2PdFNtFrShYhWfF6PL4mgWumjqVk68voKsnMk6): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179328499 (498h41m1.050411544s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179462273 (498h41m1.050545308s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179468526 (498h41m1.050551561s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179504985 (498h41m1.05058803s) [seg_pos={0 0}, upload=437697]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:10:01.179608212 (498h41m1.050691267s) [upload=437697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:10:01.179618601 (498h41m1.050701656s) [upload=437697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:10:01.469507960 (498h41m1.340591005s) [upload=437697]: Done uploading segments... Dec 26 10:10:01.469529561 (498h41m1.340612606s) [upload=437697]: Upload finished. err: <nil> Duration: 33.405659915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.X.X...X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....] duration: 6.186326633s trace: https://jaeger.dev.storj.io/trace/4316635a3e568047 FAIL (2025-12-26 10:16:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 10:13:09.299830902 (498h44m9.170913947s) [upload=437744]: Starting upload Dec 26 10:13:09.299998702 (498h44m9.171081737s) [upload=437744]: Done starting upload Dec 26 10:13:09.300042525 (498h44m9.17112557s) [upload=437744]: Uploading segments... Dec 26 10:13:09.300253667 (498h44m9.171336702s) [upload=437744]: Got next segment. Inline: false Dec 26 10:13:09.300286960 (498h44m9.171370005s) [upload=437744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:13:09.300292551 (498h44m9.171375586s) [upload=437744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:13:09.544799965 (498h44m9.41588301s) [seg_pos={0 0}, upload=437744]: Begin upload segment... Dec 26 10:13:09.544856503 (498h44m9.415939548s) [seg_pos={0 0}, upload=437744]: Done begin upload segment. Dec 26 10:13:09.544892812 (498h44m9.415975857s) [noise=true , address=145.79.227.42:4121 , piece_id=7WLJB4BYLG7SMQJF7QLMTZZIUI7NRJT4BMYA3VWDNCHULGIDL4GQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.544987383 (498h44m9.416070428s) [noise=true , address=145.79.227.45:4231 , piece_id=WBXPYOP67W6TLXUIETXGHT5KAVAETAU7JH3ZUV4TOXFU3LOIH66Q, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.545747781 (498h44m9.416830826s) [noise=true , address=145.79.227.35:4481 , piece_id=D5TEPCAZ3HBLUUL3K3QWAMF3KWEI5FE42AJ3VKJDOMPGZWTOIJVQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546444748 (498h44m9.417527783s) [noise=true , address=45.91.116.5:4300 , piece_id=NU4LU4VF5HXGHX556AO4ITALFL6EZ7RKWUCT7EZ3UEJ5RHG4QFQQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546822266 (498h44m9.417905311s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546900165 (498h44m9.4179832s) [noise=true , address=77.90.31.14:4661 , piece_id=QPDSRU3GI527Y2PPTE7DCGV4HLHCKO3FIGK2FJNDNVBMPWRN34BA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547285258 (498h44m9.418368303s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547489918 (498h44m9.418572963s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547734013 (498h44m9.418817058s) [noise=true , address=145.79.227.41:4086 , piece_id=VICDSHIIDL6W2U2Z2GMCXB2THS4W4L5HC7L5LZTNQWFSDAO6NXTA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548245546 (498h44m9.419328591s) [noise=true , address=145.79.227.50:4374 , piece_id=VBRXPTLWJWBTIVYEO7NSQBXE53IMASB3OOXAX4PRAMHECGCTSW2A, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548313476 (498h44m9.419396521s) [noise=true , address=145.79.227.38:4978 , piece_id=XEYK7X66KZVMBTRS75XRRP2J6HPWJWJ4BKHZ6BVXKD645M3PICBA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548683831 (498h44m9.419766866s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ56ZU53NLIBJMBE7JHOZGMEGOBBQ4RUYSBQ7JVRGKOLHIWX55OA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.549031253 (498h44m9.420114298s) [noise=true , address=145.79.227.49:4338 , piece_id=KIN4XQJIZV6WUXOSGI6YWOPEWJJMSBZ3CRV2346DE5HY5GTR77BQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.550888679 (498h44m9.421971724s) [noise=true , address=145.79.227.5:4223 , piece_id=4NK6SITT66ES3H3QCAKMBI3CZJJ5VWZ23V2FOVR3WQ74PTECLRCA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.551602729 (498h44m9.422685774s) [noise=true , address=145.79.227.13:4803 , piece_id=JSYV3ZL6MMH2LIRFD4OOW6ECW2XO5746UIMCLZ5NCIK3KXHEQBWA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.552717893 (498h44m9.423800938s) [noise=true , address=145.79.227.19:4078 , piece_id=FC2YJY6VFK4F3IP4VSBC35EERQMI2AIUQ5JG4K7ZS37ZZ7OSJ2OA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:11.175981496 (498h44m11.047064541s) [upload=437744]: Next returned nil segment Dec 26 10:13:11.175998008 (498h44m11.047081053s) [upload=437744]: Waiting for error group managing segments... Dec 26 10:13:11.548293842 (498h44m11.419376888s) [noise=true , address=145.79.227.41:4086 , piece_id=VICDSHIIDL6W2U2Z2GMCXB2THS4W4L5HC7L5LZTNQWFSDAO6NXTA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.550860761 (498h44m11.421943806s) [noise=true , address=145.79.227.42:4121 , piece_id=7WLJB4BYLG7SMQJF7QLMTZZIUI7NRJT4BMYA3VWDNCHULGIDL4GQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.555894426 (498h44m11.426977471s) [noise=true , address=145.79.227.49:4338 , piece_id=KIN4XQJIZV6WUXOSGI6YWOPEWJJMSBZ3CRV2346DE5HY5GTR77BQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.556737301 (498h44m11.427820356s) [noise=true , address=145.79.227.45:4231 , piece_id=WBXPYOP67W6TLXUIETXGHT5KAVAETAU7JH3ZUV4TOXFU3LOIH66Q, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559027632 (498h44m11.430110677s) [noise=true , address=145.79.227.50:4374 , piece_id=VBRXPTLWJWBTIVYEO7NSQBXE53IMASB3OOXAX4PRAMHECGCTSW2A, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559376567 (498h44m11.430459602s) [noise=true , address=145.79.227.13:4803 , piece_id=JSYV3ZL6MMH2LIRFD4OOW6ECW2XO5746UIMCLZ5NCIK3KXHEQBWA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559601324 (498h44m11.430684359s) [noise=true , address=145.79.227.19:4078 , piece_id=FC2YJY6VFK4F3IP4VSBC35EERQMI2AIUQ5JG4K7ZS37ZZ7OSJ2OA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.574555630 (498h44m11.445638705s) [noise=true , address=145.79.227.5:4223 , piece_id=4NK6SITT66ES3H3QCAKMBI3CZJJ5VWZ23V2FOVR3WQ74PTECLRCA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.576825352 (498h44m11.447908397s) [noise=true , address=145.79.227.35:4481 , piece_id=D5TEPCAZ3HBLUUL3K3QWAMF3KWEI5FE42AJ3VKJDOMPGZWTOIJVQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.582999057 (498h44m11.454082092s) [noise=true , address=145.79.227.38:4978 , piece_id=XEYK7X66KZVMBTRS75XRRP2J6HPWJWJ4BKHZ6BVXKD645M3PICBA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.607006822 (498h44m11.478089858s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ56ZU53NLIBJMBE7JHOZGMEGOBBQ4RUYSBQ7JVRGKOLHIWX55OA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.740977586 (498h44m11.612060621s) [noise=true , address=77.90.31.14:4661 , piece_id=QPDSRU3GI527Y2PPTE7DCGV4HLHCKO3FIGK2FJNDNVBMPWRN34BA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:12.961853244 (498h44m12.832936279s) [noise=true , address=45.91.116.5:4300 , piece_id=NU4LU4VF5HXGHX556AO4ITALFL6EZ7RKWUCT7EZ3UEJ5RHG4QFQQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:12.961879223 (498h44m12.832962258s) [seg_pos={0 0}, upload=437744]: Segment reached optimal threshold of 13 pieces. Dec 26 10:13:12.962102027 (498h44m12.833185062s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 26 10:13:12.962110553 (498h44m12.833193598s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962385968 (498h44m12.833469013s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 10:13:12.962395907 (498h44m12.833478942s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962559428 (498h44m12.833642473s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection Dec 26 10:13:12.962578304 (498h44m12.833661349s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962651373 (498h44m12.833734419s) [seg_pos={0 0}, upload=437744]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:13:12.962818101 (498h44m12.833901146s) [upload=437744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:13:12.962825645 (498h44m12.83390869s) [upload=437744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:13:13.256479543 (498h44m13.127562578s) [upload=437744]: Done uploading segments... Dec 26 10:13:13.256513428 (498h44m13.127596463s) [upload=437744]: Upload finished. err: <nil> Duration: 6.186354705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X......X.........X..................X......] duration: 52.60690243s trace: https://jaeger.dev.storj.io/trace/3eea9e0464bed0e7 FAIL (2025-12-26 10:15:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.333867109 (498h40m58.204950154s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.334611487 (498h40m58.205694542s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.335747089 (498h40m58.206830124s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336515553 (498h40m58.207598598s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336870569 (498h40m58.207953614s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337310336 (498h40m58.208393381s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337723112 (498h40m58.208806157s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338318425 (498h40m58.20940147s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338828727 (498h40m58.209911772s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339185056 (498h40m58.210268091s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339640863 (498h40m58.210723909s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339998325 (498h40m58.21108136s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.340445936 (498h40m58.211528981s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.341374635 (498h40m58.21245768s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:59.798768304 (498h40m59.669851359s) [upload=437697]: Next returned nil segment Dec 26 10:09:59.798783604 (498h40m59.669866649s) [upload=437697]: Waiting for error group managing segments... Dec 26 10:10:00.202844345 (498h41m0.07392739s) [noise=true , address=145.79.227.43:4175 , piece_id=5AW772QNP72OEKJTB6I4JGD4L5ONMRF5ZNXYLMKZAQDC56VOLSLQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.212993169 (498h41m0.084076204s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.224811985 (498h41m0.09589503s) [noise=true , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.239828058 (498h41m0.110911093s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.411596484 (498h41m0.282679529s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.490073348 (498h41m0.361156383s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.496009011 (498h41m0.367092056s) [noise=true , address=77.90.31.21:4904 , piece_id=NUBWDV2G25QBQKJDLQWVQG5JIE7PUW5XRHYNRCTVM3EENZPNQ6YA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.505536983 (498h41m0.376620018s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.621872607 (498h41m0.492955642s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.628231495 (498h41m0.49931453s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.634412284 (498h41m0.505495329s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.069090817 (498h41m0.940173852s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178406242 (498h41m1.049489277s) [noise=true , address=145.79.227.44:4226 , piece_id=CWN7Z7L2QIYNO75WNLF6Q4PPM3EIEAYJGYJCCY47U6NT4E4PYEVQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178763302 (498h41m1.049846347s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178785645 (498h41m1.04986869s) [seg_pos={0 0}, upload=437697]: Segment reached optimal threshold of 14 pieces. Dec 26 10:10:01.179093852 (498h41m1.050176897s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection Dec 26 10:10:01.179102047 (498h41m1.050185092s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179217547 (498h41m1.050300592s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection Dec 26 10:10:01.179223299 (498h41m1.050306334s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179319030 (498h41m1.050402075s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AZxy9wnMgVu2PdFNtFrShYhWfF6PL4mgWumjqVk68voKsnMk6): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179328499 (498h41m1.050411544s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179462273 (498h41m1.050545308s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179468526 (498h41m1.050551561s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179504985 (498h41m1.05058803s) [seg_pos={0 0}, upload=437697]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:10:01.179608212 (498h41m1.050691267s) [upload=437697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:10:01.179618601 (498h41m1.050701656s) [upload=437697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:10:01.469507960 (498h41m1.340591005s) [upload=437697]: Done uploading segments... Dec 26 10:10:01.469529561 (498h41m1.340612606s) [upload=437697]: Upload finished. err: <nil> Duration: 52.606914554s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [..X.X...X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX...] duration: 6.869601736s trace: https://jaeger.dev.storj.io/trace/4316635a3e568047 FAIL (2025-12-26 10:15:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 10:13:09.299830902 (498h44m9.170913947s) [upload=437744]: Starting upload Dec 26 10:13:09.299998702 (498h44m9.171081737s) [upload=437744]: Done starting upload Dec 26 10:13:09.300042525 (498h44m9.17112557s) [upload=437744]: Uploading segments... Dec 26 10:13:09.300253667 (498h44m9.171336702s) [upload=437744]: Got next segment. Inline: false Dec 26 10:13:09.300286960 (498h44m9.171370005s) [upload=437744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:13:09.300292551 (498h44m9.171375586s) [upload=437744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:13:09.544799965 (498h44m9.41588301s) [seg_pos={0 0}, upload=437744]: Begin upload segment... Dec 26 10:13:09.544856503 (498h44m9.415939548s) [seg_pos={0 0}, upload=437744]: Done begin upload segment. Dec 26 10:13:09.544892812 (498h44m9.415975857s) [noise=true , address=145.79.227.42:4121 , piece_id=7WLJB4BYLG7SMQJF7QLMTZZIUI7NRJT4BMYA3VWDNCHULGIDL4GQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.544987383 (498h44m9.416070428s) [noise=true , address=145.79.227.45:4231 , piece_id=WBXPYOP67W6TLXUIETXGHT5KAVAETAU7JH3ZUV4TOXFU3LOIH66Q, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.545747781 (498h44m9.416830826s) [noise=true , address=145.79.227.35:4481 , piece_id=D5TEPCAZ3HBLUUL3K3QWAMF3KWEI5FE42AJ3VKJDOMPGZWTOIJVQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546444748 (498h44m9.417527783s) [noise=true , address=45.91.116.5:4300 , piece_id=NU4LU4VF5HXGHX556AO4ITALFL6EZ7RKWUCT7EZ3UEJ5RHG4QFQQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546822266 (498h44m9.417905311s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546900165 (498h44m9.4179832s) [noise=true , address=77.90.31.14:4661 , piece_id=QPDSRU3GI527Y2PPTE7DCGV4HLHCKO3FIGK2FJNDNVBMPWRN34BA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547285258 (498h44m9.418368303s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547489918 (498h44m9.418572963s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547734013 (498h44m9.418817058s) [noise=true , address=145.79.227.41:4086 , piece_id=VICDSHIIDL6W2U2Z2GMCXB2THS4W4L5HC7L5LZTNQWFSDAO6NXTA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548245546 (498h44m9.419328591s) [noise=true , address=145.79.227.50:4374 , piece_id=VBRXPTLWJWBTIVYEO7NSQBXE53IMASB3OOXAX4PRAMHECGCTSW2A, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548313476 (498h44m9.419396521s) [noise=true , address=145.79.227.38:4978 , piece_id=XEYK7X66KZVMBTRS75XRRP2J6HPWJWJ4BKHZ6BVXKD645M3PICBA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548683831 (498h44m9.419766866s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ56ZU53NLIBJMBE7JHOZGMEGOBBQ4RUYSBQ7JVRGKOLHIWX55OA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.549031253 (498h44m9.420114298s) [noise=true , address=145.79.227.49:4338 , piece_id=KIN4XQJIZV6WUXOSGI6YWOPEWJJMSBZ3CRV2346DE5HY5GTR77BQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.550888679 (498h44m9.421971724s) [noise=true , address=145.79.227.5:4223 , piece_id=4NK6SITT66ES3H3QCAKMBI3CZJJ5VWZ23V2FOVR3WQ74PTECLRCA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.551602729 (498h44m9.422685774s) [noise=true , address=145.79.227.13:4803 , piece_id=JSYV3ZL6MMH2LIRFD4OOW6ECW2XO5746UIMCLZ5NCIK3KXHEQBWA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.552717893 (498h44m9.423800938s) [noise=true , address=145.79.227.19:4078 , piece_id=FC2YJY6VFK4F3IP4VSBC35EERQMI2AIUQ5JG4K7ZS37ZZ7OSJ2OA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:11.175981496 (498h44m11.047064541s) [upload=437744]: Next returned nil segment Dec 26 10:13:11.175998008 (498h44m11.047081053s) [upload=437744]: Waiting for error group managing segments... Dec 26 10:13:11.548293842 (498h44m11.419376888s) [noise=true , address=145.79.227.41:4086 , piece_id=VICDSHIIDL6W2U2Z2GMCXB2THS4W4L5HC7L5LZTNQWFSDAO6NXTA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.550860761 (498h44m11.421943806s) [noise=true , address=145.79.227.42:4121 , piece_id=7WLJB4BYLG7SMQJF7QLMTZZIUI7NRJT4BMYA3VWDNCHULGIDL4GQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.555894426 (498h44m11.426977471s) [noise=true , address=145.79.227.49:4338 , piece_id=KIN4XQJIZV6WUXOSGI6YWOPEWJJMSBZ3CRV2346DE5HY5GTR77BQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.556737301 (498h44m11.427820356s) [noise=true , address=145.79.227.45:4231 , piece_id=WBXPYOP67W6TLXUIETXGHT5KAVAETAU7JH3ZUV4TOXFU3LOIH66Q, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559027632 (498h44m11.430110677s) [noise=true , address=145.79.227.50:4374 , piece_id=VBRXPTLWJWBTIVYEO7NSQBXE53IMASB3OOXAX4PRAMHECGCTSW2A, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559376567 (498h44m11.430459602s) [noise=true , address=145.79.227.13:4803 , piece_id=JSYV3ZL6MMH2LIRFD4OOW6ECW2XO5746UIMCLZ5NCIK3KXHEQBWA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559601324 (498h44m11.430684359s) [noise=true , address=145.79.227.19:4078 , piece_id=FC2YJY6VFK4F3IP4VSBC35EERQMI2AIUQ5JG4K7ZS37ZZ7OSJ2OA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.574555630 (498h44m11.445638705s) [noise=true , address=145.79.227.5:4223 , piece_id=4NK6SITT66ES3H3QCAKMBI3CZJJ5VWZ23V2FOVR3WQ74PTECLRCA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.576825352 (498h44m11.447908397s) [noise=true , address=145.79.227.35:4481 , piece_id=D5TEPCAZ3HBLUUL3K3QWAMF3KWEI5FE42AJ3VKJDOMPGZWTOIJVQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.582999057 (498h44m11.454082092s) [noise=true , address=145.79.227.38:4978 , piece_id=XEYK7X66KZVMBTRS75XRRP2J6HPWJWJ4BKHZ6BVXKD645M3PICBA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.607006822 (498h44m11.478089858s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ56ZU53NLIBJMBE7JHOZGMEGOBBQ4RUYSBQ7JVRGKOLHIWX55OA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.740977586 (498h44m11.612060621s) [noise=true , address=77.90.31.14:4661 , piece_id=QPDSRU3GI527Y2PPTE7DCGV4HLHCKO3FIGK2FJNDNVBMPWRN34BA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:12.961853244 (498h44m12.832936279s) [noise=true , address=45.91.116.5:4300 , piece_id=NU4LU4VF5HXGHX556AO4ITALFL6EZ7RKWUCT7EZ3UEJ5RHG4QFQQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:12.961879223 (498h44m12.832962258s) [seg_pos={0 0}, upload=437744]: Segment reached optimal threshold of 13 pieces. Dec 26 10:13:12.962102027 (498h44m12.833185062s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 26 10:13:12.962110553 (498h44m12.833193598s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962385968 (498h44m12.833469013s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 10:13:12.962395907 (498h44m12.833478942s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962559428 (498h44m12.833642473s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection Dec 26 10:13:12.962578304 (498h44m12.833661349s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962651373 (498h44m12.833734419s) [seg_pos={0 0}, upload=437744]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:13:12.962818101 (498h44m12.833901146s) [upload=437744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:13:12.962825645 (498h44m12.83390869s) [upload=437744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:13:13.256479543 (498h44m13.127562578s) [upload=437744]: Done uploading segments... Dec 26 10:13:13.256513428 (498h44m13.127596463s) [upload=437744]: Upload finished. err: <nil> Duration: 6.869629568s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [...X.X...X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX..] duration: 5.853171107s trace: https://jaeger.dev.storj.io/trace/4316635a3e568047 FAIL (2025-12-26 10:15:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 10:13:09.299830902 (498h44m9.170913947s) [upload=437744]: Starting upload Dec 26 10:13:09.299998702 (498h44m9.171081737s) [upload=437744]: Done starting upload Dec 26 10:13:09.300042525 (498h44m9.17112557s) [upload=437744]: Uploading segments... Dec 26 10:13:09.300253667 (498h44m9.171336702s) [upload=437744]: Got next segment. Inline: false Dec 26 10:13:09.300286960 (498h44m9.171370005s) [upload=437744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:13:09.300292551 (498h44m9.171375586s) [upload=437744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:13:09.544799965 (498h44m9.41588301s) [seg_pos={0 0}, upload=437744]: Begin upload segment... Dec 26 10:13:09.544856503 (498h44m9.415939548s) [seg_pos={0 0}, upload=437744]: Done begin upload segment. Dec 26 10:13:09.544892812 (498h44m9.415975857s) [noise=true , address=145.79.227.42:4121 , piece_id=7WLJB4BYLG7SMQJF7QLMTZZIUI7NRJT4BMYA3VWDNCHULGIDL4GQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.544987383 (498h44m9.416070428s) [noise=true , address=145.79.227.45:4231 , piece_id=WBXPYOP67W6TLXUIETXGHT5KAVAETAU7JH3ZUV4TOXFU3LOIH66Q, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.545747781 (498h44m9.416830826s) [noise=true , address=145.79.227.35:4481 , piece_id=D5TEPCAZ3HBLUUL3K3QWAMF3KWEI5FE42AJ3VKJDOMPGZWTOIJVQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546444748 (498h44m9.417527783s) [noise=true , address=45.91.116.5:4300 , piece_id=NU4LU4VF5HXGHX556AO4ITALFL6EZ7RKWUCT7EZ3UEJ5RHG4QFQQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546822266 (498h44m9.417905311s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546900165 (498h44m9.4179832s) [noise=true , address=77.90.31.14:4661 , piece_id=QPDSRU3GI527Y2PPTE7DCGV4HLHCKO3FIGK2FJNDNVBMPWRN34BA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547285258 (498h44m9.418368303s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547489918 (498h44m9.418572963s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547734013 (498h44m9.418817058s) [noise=true , address=145.79.227.41:4086 , piece_id=VICDSHIIDL6W2U2Z2GMCXB2THS4W4L5HC7L5LZTNQWFSDAO6NXTA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548245546 (498h44m9.419328591s) [noise=true , address=145.79.227.50:4374 , piece_id=VBRXPTLWJWBTIVYEO7NSQBXE53IMASB3OOXAX4PRAMHECGCTSW2A, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548313476 (498h44m9.419396521s) [noise=true , address=145.79.227.38:4978 , piece_id=XEYK7X66KZVMBTRS75XRRP2J6HPWJWJ4BKHZ6BVXKD645M3PICBA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548683831 (498h44m9.419766866s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ56ZU53NLIBJMBE7JHOZGMEGOBBQ4RUYSBQ7JVRGKOLHIWX55OA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.549031253 (498h44m9.420114298s) [noise=true , address=145.79.227.49:4338 , piece_id=KIN4XQJIZV6WUXOSGI6YWOPEWJJMSBZ3CRV2346DE5HY5GTR77BQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.550888679 (498h44m9.421971724s) [noise=true , address=145.79.227.5:4223 , piece_id=4NK6SITT66ES3H3QCAKMBI3CZJJ5VWZ23V2FOVR3WQ74PTECLRCA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.551602729 (498h44m9.422685774s) [noise=true , address=145.79.227.13:4803 , piece_id=JSYV3ZL6MMH2LIRFD4OOW6ECW2XO5746UIMCLZ5NCIK3KXHEQBWA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.552717893 (498h44m9.423800938s) [noise=true , address=145.79.227.19:4078 , piece_id=FC2YJY6VFK4F3IP4VSBC35EERQMI2AIUQ5JG4K7ZS37ZZ7OSJ2OA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:11.175981496 (498h44m11.047064541s) [upload=437744]: Next returned nil segment Dec 26 10:13:11.175998008 (498h44m11.047081053s) [upload=437744]: Waiting for error group managing segments... Dec 26 10:13:11.548293842 (498h44m11.419376888s) [noise=true , address=145.79.227.41:4086 , piece_id=VICDSHIIDL6W2U2Z2GMCXB2THS4W4L5HC7L5LZTNQWFSDAO6NXTA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.550860761 (498h44m11.421943806s) [noise=true , address=145.79.227.42:4121 , piece_id=7WLJB4BYLG7SMQJF7QLMTZZIUI7NRJT4BMYA3VWDNCHULGIDL4GQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.555894426 (498h44m11.426977471s) [noise=true , address=145.79.227.49:4338 , piece_id=KIN4XQJIZV6WUXOSGI6YWOPEWJJMSBZ3CRV2346DE5HY5GTR77BQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.556737301 (498h44m11.427820356s) [noise=true , address=145.79.227.45:4231 , piece_id=WBXPYOP67W6TLXUIETXGHT5KAVAETAU7JH3ZUV4TOXFU3LOIH66Q, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559027632 (498h44m11.430110677s) [noise=true , address=145.79.227.50:4374 , piece_id=VBRXPTLWJWBTIVYEO7NSQBXE53IMASB3OOXAX4PRAMHECGCTSW2A, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559376567 (498h44m11.430459602s) [noise=true , address=145.79.227.13:4803 , piece_id=JSYV3ZL6MMH2LIRFD4OOW6ECW2XO5746UIMCLZ5NCIK3KXHEQBWA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559601324 (498h44m11.430684359s) [noise=true , address=145.79.227.19:4078 , piece_id=FC2YJY6VFK4F3IP4VSBC35EERQMI2AIUQ5JG4K7ZS37ZZ7OSJ2OA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.574555630 (498h44m11.445638705s) [noise=true , address=145.79.227.5:4223 , piece_id=4NK6SITT66ES3H3QCAKMBI3CZJJ5VWZ23V2FOVR3WQ74PTECLRCA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.576825352 (498h44m11.447908397s) [noise=true , address=145.79.227.35:4481 , piece_id=D5TEPCAZ3HBLUUL3K3QWAMF3KWEI5FE42AJ3VKJDOMPGZWTOIJVQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.582999057 (498h44m11.454082092s) [noise=true , address=145.79.227.38:4978 , piece_id=XEYK7X66KZVMBTRS75XRRP2J6HPWJWJ4BKHZ6BVXKD645M3PICBA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.607006822 (498h44m11.478089858s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ56ZU53NLIBJMBE7JHOZGMEGOBBQ4RUYSBQ7JVRGKOLHIWX55OA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.740977586 (498h44m11.612060621s) [noise=true , address=77.90.31.14:4661 , piece_id=QPDSRU3GI527Y2PPTE7DCGV4HLHCKO3FIGK2FJNDNVBMPWRN34BA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:12.961853244 (498h44m12.832936279s) [noise=true , address=45.91.116.5:4300 , piece_id=NU4LU4VF5HXGHX556AO4ITALFL6EZ7RKWUCT7EZ3UEJ5RHG4QFQQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:12.961879223 (498h44m12.832962258s) [seg_pos={0 0}, upload=437744]: Segment reached optimal threshold of 13 pieces. Dec 26 10:13:12.962102027 (498h44m12.833185062s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 26 10:13:12.962110553 (498h44m12.833193598s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962385968 (498h44m12.833469013s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 10:13:12.962395907 (498h44m12.833478942s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962559428 (498h44m12.833642473s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection Dec 26 10:13:12.962578304 (498h44m12.833661349s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962651373 (498h44m12.833734419s) [seg_pos={0 0}, upload=437744]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:13:12.962818101 (498h44m12.833901146s) [upload=437744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:13:12.962825645 (498h44m12.83390869s) [upload=437744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:13:13.256479543 (498h44m13.127562578s) [upload=437744]: Done uploading segments... Dec 26 10:13:13.256513428 (498h44m13.127596463s) [upload=437744]: Upload finished. err: <nil> Duration: 5.853222494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X......X.........X..................X.....] duration: 48.541106584s trace: https://jaeger.dev.storj.io/trace/3eea9e0464bed0e7 FAIL (2025-12-26 10:14:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.333867109 (498h40m58.204950154s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.334611487 (498h40m58.205694542s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.335747089 (498h40m58.206830124s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336515553 (498h40m58.207598598s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336870569 (498h40m58.207953614s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337310336 (498h40m58.208393381s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337723112 (498h40m58.208806157s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338318425 (498h40m58.20940147s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338828727 (498h40m58.209911772s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339185056 (498h40m58.210268091s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339640863 (498h40m58.210723909s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339998325 (498h40m58.21108136s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.340445936 (498h40m58.211528981s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.341374635 (498h40m58.21245768s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:59.798768304 (498h40m59.669851359s) [upload=437697]: Next returned nil segment Dec 26 10:09:59.798783604 (498h40m59.669866649s) [upload=437697]: Waiting for error group managing segments... Dec 26 10:10:00.202844345 (498h41m0.07392739s) [noise=true , address=145.79.227.43:4175 , piece_id=5AW772QNP72OEKJTB6I4JGD4L5ONMRF5ZNXYLMKZAQDC56VOLSLQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.212993169 (498h41m0.084076204s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.224811985 (498h41m0.09589503s) [noise=true , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.239828058 (498h41m0.110911093s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.411596484 (498h41m0.282679529s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.490073348 (498h41m0.361156383s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.496009011 (498h41m0.367092056s) [noise=true , address=77.90.31.21:4904 , piece_id=NUBWDV2G25QBQKJDLQWVQG5JIE7PUW5XRHYNRCTVM3EENZPNQ6YA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.505536983 (498h41m0.376620018s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.621872607 (498h41m0.492955642s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.628231495 (498h41m0.49931453s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.634412284 (498h41m0.505495329s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.069090817 (498h41m0.940173852s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178406242 (498h41m1.049489277s) [noise=true , address=145.79.227.44:4226 , piece_id=CWN7Z7L2QIYNO75WNLF6Q4PPM3EIEAYJGYJCCY47U6NT4E4PYEVQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178763302 (498h41m1.049846347s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178785645 (498h41m1.04986869s) [seg_pos={0 0}, upload=437697]: Segment reached optimal threshold of 14 pieces. Dec 26 10:10:01.179093852 (498h41m1.050176897s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection Dec 26 10:10:01.179102047 (498h41m1.050185092s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179217547 (498h41m1.050300592s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection Dec 26 10:10:01.179223299 (498h41m1.050306334s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179319030 (498h41m1.050402075s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AZxy9wnMgVu2PdFNtFrShYhWfF6PL4mgWumjqVk68voKsnMk6): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179328499 (498h41m1.050411544s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179462273 (498h41m1.050545308s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179468526 (498h41m1.050551561s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179504985 (498h41m1.05058803s) [seg_pos={0 0}, upload=437697]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:10:01.179608212 (498h41m1.050691267s) [upload=437697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:10:01.179618601 (498h41m1.050701656s) [upload=437697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:10:01.469507960 (498h41m1.340591005s) [upload=437697]: Done uploading segments... Dec 26 10:10:01.469529561 (498h41m1.340612606s) [upload=437697]: Upload finished. err: <nil> Duration: 48.541124818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X......X.........X..................X....] duration: 37.949534076s trace: https://jaeger.dev.storj.io/trace/3eea9e0464bed0e7 FAIL (2025-12-26 10:14:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.333867109 (498h40m58.204950154s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.334611487 (498h40m58.205694542s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.335747089 (498h40m58.206830124s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336515553 (498h40m58.207598598s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336870569 (498h40m58.207953614s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337310336 (498h40m58.208393381s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337723112 (498h40m58.208806157s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338318425 (498h40m58.20940147s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338828727 (498h40m58.209911772s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339185056 (498h40m58.210268091s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339640863 (498h40m58.210723909s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339998325 (498h40m58.21108136s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.340445936 (498h40m58.211528981s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.341374635 (498h40m58.21245768s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:59.798768304 (498h40m59.669851359s) [upload=437697]: Next returned nil segment Dec 26 10:09:59.798783604 (498h40m59.669866649s) [upload=437697]: Waiting for error group managing segments... Dec 26 10:10:00.202844345 (498h41m0.07392739s) [noise=true , address=145.79.227.43:4175 , piece_id=5AW772QNP72OEKJTB6I4JGD4L5ONMRF5ZNXYLMKZAQDC56VOLSLQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.212993169 (498h41m0.084076204s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.224811985 (498h41m0.09589503s) [noise=true , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.239828058 (498h41m0.110911093s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.411596484 (498h41m0.282679529s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.490073348 (498h41m0.361156383s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.496009011 (498h41m0.367092056s) [noise=true , address=77.90.31.21:4904 , piece_id=NUBWDV2G25QBQKJDLQWVQG5JIE7PUW5XRHYNRCTVM3EENZPNQ6YA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.505536983 (498h41m0.376620018s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.621872607 (498h41m0.492955642s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.628231495 (498h41m0.49931453s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.634412284 (498h41m0.505495329s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.069090817 (498h41m0.940173852s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178406242 (498h41m1.049489277s) [noise=true , address=145.79.227.44:4226 , piece_id=CWN7Z7L2QIYNO75WNLF6Q4PPM3EIEAYJGYJCCY47U6NT4E4PYEVQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178763302 (498h41m1.049846347s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178785645 (498h41m1.04986869s) [seg_pos={0 0}, upload=437697]: Segment reached optimal threshold of 14 pieces. Dec 26 10:10:01.179093852 (498h41m1.050176897s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection Dec 26 10:10:01.179102047 (498h41m1.050185092s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179217547 (498h41m1.050300592s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection Dec 26 10:10:01.179223299 (498h41m1.050306334s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179319030 (498h41m1.050402075s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AZxy9wnMgVu2PdFNtFrShYhWfF6PL4mgWumjqVk68voKsnMk6): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179328499 (498h41m1.050411544s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179462273 (498h41m1.050545308s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179468526 (498h41m1.050551561s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179504985 (498h41m1.05058803s) [seg_pos={0 0}, upload=437697]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:10:01.179608212 (498h41m1.050691267s) [upload=437697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:10:01.179618601 (498h41m1.050701656s) [upload=437697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:10:01.469507960 (498h41m1.340591005s) [upload=437697]: Done uploading segments... Dec 26 10:10:01.469529561 (498h41m1.340612606s) [upload=437697]: Upload finished. err: <nil> Duration: 37.949548002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [....X.X...X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX.] duration: 5.739307711s trace: https://jaeger.dev.storj.io/trace/4316635a3e568047 FAIL (2025-12-26 10:14:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 10:13:09.299830902 (498h44m9.170913947s) [upload=437744]: Starting upload Dec 26 10:13:09.299998702 (498h44m9.171081737s) [upload=437744]: Done starting upload Dec 26 10:13:09.300042525 (498h44m9.17112557s) [upload=437744]: Uploading segments... Dec 26 10:13:09.300253667 (498h44m9.171336702s) [upload=437744]: Got next segment. Inline: false Dec 26 10:13:09.300286960 (498h44m9.171370005s) [upload=437744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:13:09.300292551 (498h44m9.171375586s) [upload=437744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:13:09.544799965 (498h44m9.41588301s) [seg_pos={0 0}, upload=437744]: Begin upload segment... Dec 26 10:13:09.544856503 (498h44m9.415939548s) [seg_pos={0 0}, upload=437744]: Done begin upload segment. Dec 26 10:13:09.544892812 (498h44m9.415975857s) [noise=true , address=145.79.227.42:4121 , piece_id=7WLJB4BYLG7SMQJF7QLMTZZIUI7NRJT4BMYA3VWDNCHULGIDL4GQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.544987383 (498h44m9.416070428s) [noise=true , address=145.79.227.45:4231 , piece_id=WBXPYOP67W6TLXUIETXGHT5KAVAETAU7JH3ZUV4TOXFU3LOIH66Q, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.545747781 (498h44m9.416830826s) [noise=true , address=145.79.227.35:4481 , piece_id=D5TEPCAZ3HBLUUL3K3QWAMF3KWEI5FE42AJ3VKJDOMPGZWTOIJVQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546444748 (498h44m9.417527783s) [noise=true , address=45.91.116.5:4300 , piece_id=NU4LU4VF5HXGHX556AO4ITALFL6EZ7RKWUCT7EZ3UEJ5RHG4QFQQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546822266 (498h44m9.417905311s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546900165 (498h44m9.4179832s) [noise=true , address=77.90.31.14:4661 , piece_id=QPDSRU3GI527Y2PPTE7DCGV4HLHCKO3FIGK2FJNDNVBMPWRN34BA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547285258 (498h44m9.418368303s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547489918 (498h44m9.418572963s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547734013 (498h44m9.418817058s) [noise=true , address=145.79.227.41:4086 , piece_id=VICDSHIIDL6W2U2Z2GMCXB2THS4W4L5HC7L5LZTNQWFSDAO6NXTA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548245546 (498h44m9.419328591s) [noise=true , address=145.79.227.50:4374 , piece_id=VBRXPTLWJWBTIVYEO7NSQBXE53IMASB3OOXAX4PRAMHECGCTSW2A, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548313476 (498h44m9.419396521s) [noise=true , address=145.79.227.38:4978 , piece_id=XEYK7X66KZVMBTRS75XRRP2J6HPWJWJ4BKHZ6BVXKD645M3PICBA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548683831 (498h44m9.419766866s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ56ZU53NLIBJMBE7JHOZGMEGOBBQ4RUYSBQ7JVRGKOLHIWX55OA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.549031253 (498h44m9.420114298s) [noise=true , address=145.79.227.49:4338 , piece_id=KIN4XQJIZV6WUXOSGI6YWOPEWJJMSBZ3CRV2346DE5HY5GTR77BQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.550888679 (498h44m9.421971724s) [noise=true , address=145.79.227.5:4223 , piece_id=4NK6SITT66ES3H3QCAKMBI3CZJJ5VWZ23V2FOVR3WQ74PTECLRCA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.551602729 (498h44m9.422685774s) [noise=true , address=145.79.227.13:4803 , piece_id=JSYV3ZL6MMH2LIRFD4OOW6ECW2XO5746UIMCLZ5NCIK3KXHEQBWA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.552717893 (498h44m9.423800938s) [noise=true , address=145.79.227.19:4078 , piece_id=FC2YJY6VFK4F3IP4VSBC35EERQMI2AIUQ5JG4K7ZS37ZZ7OSJ2OA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:11.175981496 (498h44m11.047064541s) [upload=437744]: Next returned nil segment Dec 26 10:13:11.175998008 (498h44m11.047081053s) [upload=437744]: Waiting for error group managing segments... Dec 26 10:13:11.548293842 (498h44m11.419376888s) [noise=true , address=145.79.227.41:4086 , piece_id=VICDSHIIDL6W2U2Z2GMCXB2THS4W4L5HC7L5LZTNQWFSDAO6NXTA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.550860761 (498h44m11.421943806s) [noise=true , address=145.79.227.42:4121 , piece_id=7WLJB4BYLG7SMQJF7QLMTZZIUI7NRJT4BMYA3VWDNCHULGIDL4GQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.555894426 (498h44m11.426977471s) [noise=true , address=145.79.227.49:4338 , piece_id=KIN4XQJIZV6WUXOSGI6YWOPEWJJMSBZ3CRV2346DE5HY5GTR77BQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.556737301 (498h44m11.427820356s) [noise=true , address=145.79.227.45:4231 , piece_id=WBXPYOP67W6TLXUIETXGHT5KAVAETAU7JH3ZUV4TOXFU3LOIH66Q, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559027632 (498h44m11.430110677s) [noise=true , address=145.79.227.50:4374 , piece_id=VBRXPTLWJWBTIVYEO7NSQBXE53IMASB3OOXAX4PRAMHECGCTSW2A, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559376567 (498h44m11.430459602s) [noise=true , address=145.79.227.13:4803 , piece_id=JSYV3ZL6MMH2LIRFD4OOW6ECW2XO5746UIMCLZ5NCIK3KXHEQBWA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559601324 (498h44m11.430684359s) [noise=true , address=145.79.227.19:4078 , piece_id=FC2YJY6VFK4F3IP4VSBC35EERQMI2AIUQ5JG4K7ZS37ZZ7OSJ2OA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.574555630 (498h44m11.445638705s) [noise=true , address=145.79.227.5:4223 , piece_id=4NK6SITT66ES3H3QCAKMBI3CZJJ5VWZ23V2FOVR3WQ74PTECLRCA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.576825352 (498h44m11.447908397s) [noise=true , address=145.79.227.35:4481 , piece_id=D5TEPCAZ3HBLUUL3K3QWAMF3KWEI5FE42AJ3VKJDOMPGZWTOIJVQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.582999057 (498h44m11.454082092s) [noise=true , address=145.79.227.38:4978 , piece_id=XEYK7X66KZVMBTRS75XRRP2J6HPWJWJ4BKHZ6BVXKD645M3PICBA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.607006822 (498h44m11.478089858s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ56ZU53NLIBJMBE7JHOZGMEGOBBQ4RUYSBQ7JVRGKOLHIWX55OA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.740977586 (498h44m11.612060621s) [noise=true , address=77.90.31.14:4661 , piece_id=QPDSRU3GI527Y2PPTE7DCGV4HLHCKO3FIGK2FJNDNVBMPWRN34BA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:12.961853244 (498h44m12.832936279s) [noise=true , address=45.91.116.5:4300 , piece_id=NU4LU4VF5HXGHX556AO4ITALFL6EZ7RKWUCT7EZ3UEJ5RHG4QFQQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:12.961879223 (498h44m12.832962258s) [seg_pos={0 0}, upload=437744]: Segment reached optimal threshold of 13 pieces. Dec 26 10:13:12.962102027 (498h44m12.833185062s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 26 10:13:12.962110553 (498h44m12.833193598s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962385968 (498h44m12.833469013s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 10:13:12.962395907 (498h44m12.833478942s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962559428 (498h44m12.833642473s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection Dec 26 10:13:12.962578304 (498h44m12.833661349s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962651373 (498h44m12.833734419s) [seg_pos={0 0}, upload=437744]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:13:12.962818101 (498h44m12.833901146s) [upload=437744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:13:12.962825645 (498h44m12.83390869s) [upload=437744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:13:13.256479543 (498h44m13.127562578s) [upload=437744]: Done uploading segments... Dec 26 10:13:13.256513428 (498h44m13.127596463s) [upload=437744]: Upload finished. err: <nil> Duration: 5.739341657s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.....X.X...X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX] duration: 30.005825366s trace: https://jaeger.dev.storj.io/trace/4316635a3e568047 FAIL (2025-12-26 10:13:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 10:13:09.299830902 (498h44m9.170913947s) [upload=437744]: Starting upload Dec 26 10:13:09.299998702 (498h44m9.171081737s) [upload=437744]: Done starting upload Dec 26 10:13:09.300042525 (498h44m9.17112557s) [upload=437744]: Uploading segments... Dec 26 10:13:09.300253667 (498h44m9.171336702s) [upload=437744]: Got next segment. Inline: false Dec 26 10:13:09.300286960 (498h44m9.171370005s) [upload=437744]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:13:09.300292551 (498h44m9.171375586s) [upload=437744]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:13:09.544799965 (498h44m9.41588301s) [seg_pos={0 0}, upload=437744]: Begin upload segment... Dec 26 10:13:09.544856503 (498h44m9.415939548s) [seg_pos={0 0}, upload=437744]: Done begin upload segment. Dec 26 10:13:09.544892812 (498h44m9.415975857s) [noise=true , address=145.79.227.42:4121 , piece_id=7WLJB4BYLG7SMQJF7QLMTZZIUI7NRJT4BMYA3VWDNCHULGIDL4GQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.544987383 (498h44m9.416070428s) [noise=true , address=145.79.227.45:4231 , piece_id=WBXPYOP67W6TLXUIETXGHT5KAVAETAU7JH3ZUV4TOXFU3LOIH66Q, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.545747781 (498h44m9.416830826s) [noise=true , address=145.79.227.35:4481 , piece_id=D5TEPCAZ3HBLUUL3K3QWAMF3KWEI5FE42AJ3VKJDOMPGZWTOIJVQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546444748 (498h44m9.417527783s) [noise=true , address=45.91.116.5:4300 , piece_id=NU4LU4VF5HXGHX556AO4ITALFL6EZ7RKWUCT7EZ3UEJ5RHG4QFQQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546822266 (498h44m9.417905311s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.546900165 (498h44m9.4179832s) [noise=true , address=77.90.31.14:4661 , piece_id=QPDSRU3GI527Y2PPTE7DCGV4HLHCKO3FIGK2FJNDNVBMPWRN34BA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547285258 (498h44m9.418368303s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547489918 (498h44m9.418572963s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.547734013 (498h44m9.418817058s) [noise=true , address=145.79.227.41:4086 , piece_id=VICDSHIIDL6W2U2Z2GMCXB2THS4W4L5HC7L5LZTNQWFSDAO6NXTA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548245546 (498h44m9.419328591s) [noise=true , address=145.79.227.50:4374 , piece_id=VBRXPTLWJWBTIVYEO7NSQBXE53IMASB3OOXAX4PRAMHECGCTSW2A, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548313476 (498h44m9.419396521s) [noise=true , address=145.79.227.38:4978 , piece_id=XEYK7X66KZVMBTRS75XRRP2J6HPWJWJ4BKHZ6BVXKD645M3PICBA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.548683831 (498h44m9.419766866s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ56ZU53NLIBJMBE7JHOZGMEGOBBQ4RUYSBQ7JVRGKOLHIWX55OA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.549031253 (498h44m9.420114298s) [noise=true , address=145.79.227.49:4338 , piece_id=KIN4XQJIZV6WUXOSGI6YWOPEWJJMSBZ3CRV2346DE5HY5GTR77BQ, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.550888679 (498h44m9.421971724s) [noise=true , address=145.79.227.5:4223 , piece_id=4NK6SITT66ES3H3QCAKMBI3CZJJ5VWZ23V2FOVR3WQ74PTECLRCA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.551602729 (498h44m9.422685774s) [noise=true , address=145.79.227.13:4803 , piece_id=JSYV3ZL6MMH2LIRFD4OOW6ECW2XO5746UIMCLZ5NCIK3KXHEQBWA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:09.552717893 (498h44m9.423800938s) [noise=true , address=145.79.227.19:4078 , piece_id=FC2YJY6VFK4F3IP4VSBC35EERQMI2AIUQ5JG4K7ZS37ZZ7OSJ2OA, seg_pos={0 0}, upload=437744]: Uploading piece... Dec 26 10:13:11.175981496 (498h44m11.047064541s) [upload=437744]: Next returned nil segment Dec 26 10:13:11.175998008 (498h44m11.047081053s) [upload=437744]: Waiting for error group managing segments... Dec 26 10:13:11.548293842 (498h44m11.419376888s) [noise=true , address=145.79.227.41:4086 , piece_id=VICDSHIIDL6W2U2Z2GMCXB2THS4W4L5HC7L5LZTNQWFSDAO6NXTA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.550860761 (498h44m11.421943806s) [noise=true , address=145.79.227.42:4121 , piece_id=7WLJB4BYLG7SMQJF7QLMTZZIUI7NRJT4BMYA3VWDNCHULGIDL4GQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.555894426 (498h44m11.426977471s) [noise=true , address=145.79.227.49:4338 , piece_id=KIN4XQJIZV6WUXOSGI6YWOPEWJJMSBZ3CRV2346DE5HY5GTR77BQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.556737301 (498h44m11.427820356s) [noise=true , address=145.79.227.45:4231 , piece_id=WBXPYOP67W6TLXUIETXGHT5KAVAETAU7JH3ZUV4TOXFU3LOIH66Q, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559027632 (498h44m11.430110677s) [noise=true , address=145.79.227.50:4374 , piece_id=VBRXPTLWJWBTIVYEO7NSQBXE53IMASB3OOXAX4PRAMHECGCTSW2A, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559376567 (498h44m11.430459602s) [noise=true , address=145.79.227.13:4803 , piece_id=JSYV3ZL6MMH2LIRFD4OOW6ECW2XO5746UIMCLZ5NCIK3KXHEQBWA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.559601324 (498h44m11.430684359s) [noise=true , address=145.79.227.19:4078 , piece_id=FC2YJY6VFK4F3IP4VSBC35EERQMI2AIUQ5JG4K7ZS37ZZ7OSJ2OA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.574555630 (498h44m11.445638705s) [noise=true , address=145.79.227.5:4223 , piece_id=4NK6SITT66ES3H3QCAKMBI3CZJJ5VWZ23V2FOVR3WQ74PTECLRCA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.576825352 (498h44m11.447908397s) [noise=true , address=145.79.227.35:4481 , piece_id=D5TEPCAZ3HBLUUL3K3QWAMF3KWEI5FE42AJ3VKJDOMPGZWTOIJVQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.582999057 (498h44m11.454082092s) [noise=true , address=145.79.227.38:4978 , piece_id=XEYK7X66KZVMBTRS75XRRP2J6HPWJWJ4BKHZ6BVXKD645M3PICBA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.607006822 (498h44m11.478089858s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ56ZU53NLIBJMBE7JHOZGMEGOBBQ4RUYSBQ7JVRGKOLHIWX55OA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:11.740977586 (498h44m11.612060621s) [noise=true , address=77.90.31.14:4661 , piece_id=QPDSRU3GI527Y2PPTE7DCGV4HLHCKO3FIGK2FJNDNVBMPWRN34BA, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:12.961853244 (498h44m12.832936279s) [noise=true , address=45.91.116.5:4300 , piece_id=NU4LU4VF5HXGHX556AO4ITALFL6EZ7RKWUCT7EZ3UEJ5RHG4QFQQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: <nil> Dec 26 10:13:12.961879223 (498h44m12.832962258s) [seg_pos={0 0}, upload=437744]: Segment reached optimal threshold of 13 pieces. Dec 26 10:13:12.962102027 (498h44m12.833185062s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 26 10:13:12.962110553 (498h44m12.833193598s) [noise=true , address=77.90.31.11:4555 , piece_id=OYGSCCV3MYPYUROQ7TY4P7TJZEXZEH3CUFAN3D4ZO5LCGAY36FRQ, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962385968 (498h44m12.833469013s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 10:13:12.962395907 (498h44m12.833478942s) [noise=true , address=77.90.31.10:4519 , piece_id=KYTQID5DAKFCR3ODOF7LH5GBGEMZYBJEB4QNMS5NI4VZW3FQVQTQ, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962559428 (498h44m12.833642473s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48126->77.90.31.18:4808: use of closed network connection Dec 26 10:13:12.962578304 (498h44m12.833661349s) [noise=true , address=77.90.31.18:4808 , piece_id=IZX3D43RKG5KA3AN3DSEXOIJ3JSYZLVCGVI7SCDUAH66LH5CJX5Q, seg_pos={0 0}, upload=437744]: Long tail cancellation due to optimal threshold Dec 26 10:13:12.962651373 (498h44m12.833734419s) [seg_pos={0 0}, upload=437744]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:13:12.962818101 (498h44m12.833901146s) [upload=437744]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:13:12.962825645 (498h44m12.83390869s) [upload=437744]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:13:13.256479543 (498h44m13.127562578s) [upload=437744]: Done uploading segments... Dec 26 10:13:13.256513428 (498h44m13.127596463s) [upload=437744]: Upload finished. err: <nil> Duration: 30.005843681s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X......X.........X..................X...] duration: 50.163429457s trace: https://jaeger.dev.storj.io/trace/3eea9e0464bed0e7 FAIL (2025-12-26 10:13:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.333867109 (498h40m58.204950154s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.334611487 (498h40m58.205694542s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.335747089 (498h40m58.206830124s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336515553 (498h40m58.207598598s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336870569 (498h40m58.207953614s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337310336 (498h40m58.208393381s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337723112 (498h40m58.208806157s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338318425 (498h40m58.20940147s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338828727 (498h40m58.209911772s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339185056 (498h40m58.210268091s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339640863 (498h40m58.210723909s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339998325 (498h40m58.21108136s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.340445936 (498h40m58.211528981s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.341374635 (498h40m58.21245768s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:59.798768304 (498h40m59.669851359s) [upload=437697]: Next returned nil segment Dec 26 10:09:59.798783604 (498h40m59.669866649s) [upload=437697]: Waiting for error group managing segments... Dec 26 10:10:00.202844345 (498h41m0.07392739s) [noise=true , address=145.79.227.43:4175 , piece_id=5AW772QNP72OEKJTB6I4JGD4L5ONMRF5ZNXYLMKZAQDC56VOLSLQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.212993169 (498h41m0.084076204s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.224811985 (498h41m0.09589503s) [noise=true , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.239828058 (498h41m0.110911093s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.411596484 (498h41m0.282679529s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.490073348 (498h41m0.361156383s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.496009011 (498h41m0.367092056s) [noise=true , address=77.90.31.21:4904 , piece_id=NUBWDV2G25QBQKJDLQWVQG5JIE7PUW5XRHYNRCTVM3EENZPNQ6YA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.505536983 (498h41m0.376620018s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.621872607 (498h41m0.492955642s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.628231495 (498h41m0.49931453s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.634412284 (498h41m0.505495329s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.069090817 (498h41m0.940173852s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178406242 (498h41m1.049489277s) [noise=true , address=145.79.227.44:4226 , piece_id=CWN7Z7L2QIYNO75WNLF6Q4PPM3EIEAYJGYJCCY47U6NT4E4PYEVQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178763302 (498h41m1.049846347s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178785645 (498h41m1.04986869s) [seg_pos={0 0}, upload=437697]: Segment reached optimal threshold of 14 pieces. Dec 26 10:10:01.179093852 (498h41m1.050176897s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection Dec 26 10:10:01.179102047 (498h41m1.050185092s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179217547 (498h41m1.050300592s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection Dec 26 10:10:01.179223299 (498h41m1.050306334s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179319030 (498h41m1.050402075s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AZxy9wnMgVu2PdFNtFrShYhWfF6PL4mgWumjqVk68voKsnMk6): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179328499 (498h41m1.050411544s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179462273 (498h41m1.050545308s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179468526 (498h41m1.050551561s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179504985 (498h41m1.05058803s) [seg_pos={0 0}, upload=437697]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:10:01.179608212 (498h41m1.050691267s) [upload=437697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:10:01.179618601 (498h41m1.050701656s) [upload=437697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:10:01.469507960 (498h41m1.340591005s) [upload=437697]: Done uploading segments... Dec 26 10:10:01.469529561 (498h41m1.340612606s) [upload=437697]: Upload finished. err: <nil> Duration: 50.163444326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X.....X.X...X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.X] duration: 30.003724182s trace: https://jaeger.dev.storj.io/trace/370d9aca6d63c2b7 FAIL (2025-12-26 10:12:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 2KCDGL22QJIALYHNUTQ2YNSRARCTX3PDJVYX7BK7EAXP3SDQHF2Q) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: UNWSU5D36Q64RYWH7RNHMVN22CSR6X2XUXOKIOBVZQIXAUGQXMVQ) context deadline exceeded ======LOGS====== 26 10:12:23.857399180 (498h43m23.728482215s) [upload=437734]: Starting upload Dec 26 10:12:23.857454897 (498h43m23.728537932s) [upload=437734]: Done starting upload Dec 26 10:12:23.857629790 (498h43m23.728712835s) [upload=437734]: Uploading segments... Dec 26 10:12:23.858531145 (498h43m23.72961419s) [upload=437734]: Got next segment. Inline: false Dec 26 10:12:23.858590659 (498h43m23.729673704s) [upload=437734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:12:23.858596780 (498h43m23.729679815s) [upload=437734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:12:24.104683382 (498h43m23.975766417s) [seg_pos={0 0}, upload=437734]: Begin upload segment... Dec 26 10:12:24.104731453 (498h43m23.975814488s) [seg_pos={0 0}, upload=437734]: Done begin upload segment. Dec 26 10:12:24.104805735 (498h43m23.97588877s) [noise=true , address=145.79.227.7:4295 , piece_id=65TGAJNRWKQH3L2RCTADCJVXPC6UB5ZD7EWNJMVE2536S5WC2EEA, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.104813379 (498h43m23.975896414s) [noise=true , address=145.79.227.52:4446 , piece_id=54FNU5VFEK4GTZCSYWB63IVLEG25QFCA4DCXVEA6DTBJABAIT3MA, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.105384146 (498h43m23.976467192s) [noise=true , address=145.79.227.19:4077 , piece_id=QD3T2QKCAYAEDDZ3SB6PMJUUKO4UN5BP64X53NOVBRFZRXJ4TW3A, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.105734243 (498h43m23.976817288s) [noise=true , address=45.91.116.10:4083 , piece_id=UNWSU5D36Q64RYWH7RNHMVN22CSR6X2XUXOKIOBVZQIXAUGQXMVQ, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.106014978 (498h43m23.977098023s) [noise=true , address=145.79.227.47:4265 , piece_id=4ARDTMMRGPGEGGBL7KH4KUFGGETT7AVX7EELDJUZ7Y6HVM2IOS3Q, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.106101231 (498h43m23.977184276s) [noise=true , address=145.79.227.37:4567 , piece_id=5IKZNYBGY3FRZNXEMAK4XV2JMYQHXDPIFPKFPCAGCLN2RQSAPHLQ, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.106475645 (498h43m23.97755869s) [noise=true , address=45.91.116.7:4011 , piece_id=T52KOKRSE6M53VE2QULIWMW3FAPLQON5QJ5GWIJVWF35MYH4VGOQ, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.107776351 (498h43m23.978859386s) [noise=true , address=145.79.227.11:4730 , piece_id=LCNHXISC5QS64DPONRQEJAWYA2K4NHZWVZVC53E5L6NIHUOESQ6Q, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.108262086 (498h43m23.979345131s) [noise=true , address=77.90.31.11:4554 , piece_id=KI7RC5VGZWJS4HOLXMF25AZGD6SKZ5PB7JSU7SG4WBTEVSP75JWQ, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.108276073 (498h43m23.979359108s) [noise=true , address=145.79.227.43:4158 , piece_id=COTH5T7K6K6MSJCBJW6SNGT3D6HJIJRWISWHVOSLXLOEK5PJUPVQ, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.108897577 (498h43m23.979980612s) [noise=true , address=45.91.116.14:4192 , piece_id=2KCDGL22QJIALYHNUTQ2YNSRARCTX3PDJVYX7BK7EAXP3SDQHF2Q, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.109315863 (498h43m23.980398908s) [noise=true , address=45.91.116.4:4940 , piece_id=4NFQTZAGE3DMQNZRVKCAS3Z2P43GG5SMJKUF4DMFD3O3VMOECF5A, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.109746563 (498h43m23.980829608s) [noise=true , address=145.79.227.16:4910 , piece_id=FYGV6OQJQPIHWWAIUVV65NZAI6AOSXAPRS6ZPVPO4J5UEVJ5JGNQ, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.110076371 (498h43m23.981159417s) [noise=true , address=145.79.227.45:4231 , piece_id=VUX5CXFCWSWLSKFAUU67VGFI6OPOREJCCTB3GCEQYKBQOUWKSFFA, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.110285209 (498h43m23.981368244s) [noise=true , address=145.79.227.44:4193 , piece_id=4YAEBMETMD7EP7HHBE2HRZGQUKQDVCYJ33TLEXDPSTJTDCQJ5PSQ, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:24.111588290 (498h43m23.982671325s) [noise=true , address=145.79.227.48:4301 , piece_id=RZ4TRBEGIPXWWPIIDQWTEXBJA4RQXHB6BZL4EAZKL4SVUKS2G6JA, seg_pos={0 0}, upload=437734]: Uploading piece... Dec 26 10:12:25.812517482 (498h43m25.683600527s) [upload=437734]: Next returned nil segment Dec 26 10:12:25.812530988 (498h43m25.683614023s) [upload=437734]: Waiting for error group managing segments... Dec 26 10:12:26.102491362 (498h43m25.973574407s) [noise=true , address=145.79.227.45:4231 , piece_id=VUX5CXFCWSWLSKFAUU67VGFI6OPOREJCCTB3GCEQYKBQOUWKSFFA, seg_pos={0 0}, upload=437734]: Done uploading piece. err: <nil> Dec 26 10:12:26.111643900 (498h43m25.982726945s) [noise=true , address=145.79.227.7:4295 , piece_id=65TGAJNRWKQH3L2RCTADCJVXPC6UB5ZD7EWNJMVE2536S5WC2EEA, seg_pos={0 0}, upload=437734]: Done uploading piece. err: <nil> Dec 26 10:12:26.115561029 (498h43m25.986644074s) [noise=true , address=145.79.227.16:4910 , piece_id=FYGV6OQJQPIHWWAIUVV65NZAI6AOSXAPRS6ZPVPO4J5UEVJ5JGNQ, seg_pos={0 0}, upload=437734]: Done uploading piece. err: <nil> Dec 26 10:12:26.119547169 (498h43m25.990630214s) [noise=true , address=145.79.227.43:4158 , piece_id=COTH5T7K6K6MSJCBJW6SNGT3D6HJIJRWISWHVOSLXLOEK5PJUPVQ, seg_pos={0 0}, upload=437734]: Done uploading piece. err: <nil> Dec 26 10:12:26.120235690 (498h43m25.991318726s) [noise=true , address=145.79.227.37:4567 , piece_id=5IKZNYBGY3FRZNXEMAK4XV2JMYQHXDPIFPKFPCAGCLN2RQSAPHLQ, seg_pos={0 0}, upload=437734]: Done uploading piece. err: <nil> Dec 26 10:12:26.129522755 (498h43m26.00060579s) [noise=true , address=145.79.227.11:4730 , piece_id=LCNHXISC5QS64DPONRQEJAWYA2K4NHZWVZVC53E5L6NIHUOESQ6Q, seg_pos={0 0}, upload=437734]: Done uploading piece. err: <nil> Dec 26 10:12:26.135878898 (498h43m26.006961944s) [noise=true , address=145.79.227.52:4446 , piece_id=54FNU5VFEK4GTZCSYWB63IVLEG25QFCA4DCXVEA6DTBJABAIT3MA, seg_pos={0 0}, upload=437734]: Done uploading piece. err: <nil> Dec 26 10:12:26.136489451 (498h43m26.007572486s) [noise=true , address=145.79.227.48:4301 , piece_id=RZ4TRBEGIPXWWPIIDQWTEXBJA4RQXHB6BZL4EAZKL4SVUKS2G6JA, seg_pos={0 0}, upload=437734]: Done uploading piece. err: <nil> Dec 26 10:12:26.141886598 (498h43m26.012969633s) [noise=true , address=145.79.227.19:4077 , piece_id=QD3T2QKCAYAEDDZ3SB6PMJUUKO4UN5BP64X53NOVBRFZRXJ4TW3A, seg_pos={0 0}, upload=437734]: Done uploading piece. err: <nil> Dec 26 10:12:26.150740127 (498h43m26.021823162s) [noise=true , address=145.79.227.47:4265 , piece_id=4ARDTMMRGPGEGGBL7KH4KUFGGETT7AVX7EELDJUZ7Y6HVM2IOS3Q, seg_pos={0 0}, upload=437734]: Done uploading piece. err: <nil> Dec 26 10:12:26.450851710 (498h43m26.321934745s) [noise=true , address=45.91.116.10:4083 , piece_id=UNWSU5D36Q64RYWH7RNHMVN22CSR6X2XUXOKIOBVZQIXAUGQXMVQ, seg_pos={0 0}, upload=437734]: Done uploading piece. err: <nil> Dec 26 10:12:26.732816520 (498h43m26.603899555s) [noise=true , address=45.91.116.14:4192 , piece_id=2KCDGL22QJIALYHNUTQ2YNSRARCTX3PDJVYX7BK7EAXP3SDQHF2Q, seg_pos={0 0}, upload=437734]: Done uploading piece. err: <nil> Dec 26 10:12:27.361029176 (498h43m27.232112221s) [noise=true , address=45.91.116.4:4940 , piece_id=4NFQTZAGE3DMQNZRVKCAS3Z2P43GG5SMJKUF4DMFD3O3VMOECF5A, seg_pos={0 0}, upload=437734]: Done uploading piece. err: <nil> Dec 26 10:12:27.361053521 (498h43m27.232136566s) [seg_pos={0 0}, upload=437734]: Segment reached optimal threshold of 13 pieces. Dec 26 10:12:27.361346630 (498h43m27.232429675s) [noise=true , address=77.90.31.11:4554 , piece_id=KI7RC5VGZWJS4HOLXMF25AZGD6SKZ5PB7JSU7SG4WBTEVSP75JWQ, seg_pos={0 0}, upload=437734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:55300->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:55300->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55300->77.90.31.11:4554: use of closed network connection Dec 26 10:12:27.361355907 (498h43m27.232438952s) [noise=true , address=77.90.31.11:4554 , piece_id=KI7RC5VGZWJS4HOLXMF25AZGD6SKZ5PB7JSU7SG4WBTEVSP75JWQ, seg_pos={0 0}, upload=437734]: Long tail cancellation due to optimal threshold Dec 26 10:12:27.361454746 (498h43m27.232537791s) [noise=true , address=145.79.227.44:4193 , piece_id=4YAEBMETMD7EP7HHBE2HRZGQUKQDVCYJ33TLEXDPSTJTDCQJ5PSQ, seg_pos={0 0}, upload=437734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 10:12:27.361474663 (498h43m27.232557718s) [noise=true , address=145.79.227.44:4193 , piece_id=4YAEBMETMD7EP7HHBE2HRZGQUKQDVCYJ33TLEXDPSTJTDCQJ5PSQ, seg_pos={0 0}, upload=437734]: Long tail cancellation due to optimal threshold Dec 26 10:12:27.361680786 (498h43m27.232763821s) [noise=true , address=45.91.116.7:4011 , piece_id=T52KOKRSE6M53VE2QULIWMW3FAPLQON5QJ5GWIJVWF35MYH4VGOQ, seg_pos={0 0}, upload=437734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 10:12:27.361690875 (498h43m27.232773921s) [noise=true , address=45.91.116.7:4011 , piece_id=T52KOKRSE6M53VE2QULIWMW3FAPLQON5QJ5GWIJVWF35MYH4VGOQ, seg_pos={0 0}, upload=437734]: Long tail cancellation due to optimal threshold Dec 26 10:12:27.361715391 (498h43m27.232798427s) [seg_pos={0 0}, upload=437734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:12:27.361885526 (498h43m27.232968571s) [upload=437734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:12:27.361892769 (498h43m27.232975814s) [upload=437734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:12:27.654066088 (498h43m27.525149133s) [upload=437734]: Done uploading segments... Dec 26 10:12:27.654095254 (498h43m27.525178299s) [upload=437734]: Upload finished. err: <nil> Duration: 30.003745783s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X......X.........X..................X..] duration: 50.700686156s trace: https://jaeger.dev.storj.io/trace/3eea9e0464bed0e7 FAIL (2025-12-26 10:12:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.333867109 (498h40m58.204950154s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.334611487 (498h40m58.205694542s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.335747089 (498h40m58.206830124s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336515553 (498h40m58.207598598s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336870569 (498h40m58.207953614s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337310336 (498h40m58.208393381s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337723112 (498h40m58.208806157s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338318425 (498h40m58.20940147s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338828727 (498h40m58.209911772s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339185056 (498h40m58.210268091s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339640863 (498h40m58.210723909s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339998325 (498h40m58.21108136s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.340445936 (498h40m58.211528981s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.341374635 (498h40m58.21245768s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:59.798768304 (498h40m59.669851359s) [upload=437697]: Next returned nil segment Dec 26 10:09:59.798783604 (498h40m59.669866649s) [upload=437697]: Waiting for error group managing segments... Dec 26 10:10:00.202844345 (498h41m0.07392739s) [noise=true , address=145.79.227.43:4175 , piece_id=5AW772QNP72OEKJTB6I4JGD4L5ONMRF5ZNXYLMKZAQDC56VOLSLQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.212993169 (498h41m0.084076204s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.224811985 (498h41m0.09589503s) [noise=true , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.239828058 (498h41m0.110911093s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.411596484 (498h41m0.282679529s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.490073348 (498h41m0.361156383s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.496009011 (498h41m0.367092056s) [noise=true , address=77.90.31.21:4904 , piece_id=NUBWDV2G25QBQKJDLQWVQG5JIE7PUW5XRHYNRCTVM3EENZPNQ6YA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.505536983 (498h41m0.376620018s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.621872607 (498h41m0.492955642s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.628231495 (498h41m0.49931453s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.634412284 (498h41m0.505495329s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.069090817 (498h41m0.940173852s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178406242 (498h41m1.049489277s) [noise=true , address=145.79.227.44:4226 , piece_id=CWN7Z7L2QIYNO75WNLF6Q4PPM3EIEAYJGYJCCY47U6NT4E4PYEVQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178763302 (498h41m1.049846347s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178785645 (498h41m1.04986869s) [seg_pos={0 0}, upload=437697]: Segment reached optimal threshold of 14 pieces. Dec 26 10:10:01.179093852 (498h41m1.050176897s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection Dec 26 10:10:01.179102047 (498h41m1.050185092s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179217547 (498h41m1.050300592s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection Dec 26 10:10:01.179223299 (498h41m1.050306334s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179319030 (498h41m1.050402075s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AZxy9wnMgVu2PdFNtFrShYhWfF6PL4mgWumjqVk68voKsnMk6): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179328499 (498h41m1.050411544s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179462273 (498h41m1.050545308s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179468526 (498h41m1.050551561s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179504985 (498h41m1.05058803s) [seg_pos={0 0}, upload=437697]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:10:01.179608212 (498h41m1.050691267s) [upload=437697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:10:01.179618601 (498h41m1.050701656s) [upload=437697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:10:01.469507960 (498h41m1.340591005s) [upload=437697]: Done uploading segments... Dec 26 10:10:01.469529561 (498h41m1.340612606s) [upload=437697]: Upload finished. err: <nil> Duration: 50.700701424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X.....X.X...X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X.] duration: 6.161414907s trace: https://jaeger.dev.storj.io/trace/97d1e0d9e37debb FAIL (2025-12-26 10:11:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 10:11:15.640525945 (498h42m15.51160899s) [upload=437717]: Starting upload Dec 26 10:11:15.640625635 (498h42m15.51170867s) [upload=437717]: Done starting upload Dec 26 10:11:15.640690639 (498h42m15.511773674s) [upload=437717]: Uploading segments... Dec 26 10:11:15.640875571 (498h42m15.511958616s) [upload=437717]: Got next segment. Inline: false Dec 26 10:11:15.640913042 (498h42m15.511996077s) [upload=437717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:11:15.640919003 (498h42m15.512002038s) [upload=437717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:11:15.887069547 (498h42m15.758152582s) [seg_pos={0 0}, upload=437717]: Begin upload segment... Dec 26 10:11:15.887113841 (498h42m15.758196876s) [seg_pos={0 0}, upload=437717]: Done begin upload segment. Dec 26 10:11:15.887139410 (498h42m15.758222455s) [noise=true , address=45.91.116.7:4009 , piece_id=UPXB3KAIFMWIK72GLQ3QY7QG7K2D3OPH2L3Z4OCDXCYQHHSHH67A, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.887183634 (498h42m15.758266679s) [noise=true , address=77.90.31.16:4735 , piece_id=VOZWPS377M4XODPZQY4KJKJGBDDRMBSXV6DM627MCQ5OF3IC25FQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.887780100 (498h42m15.758863135s) [noise=true , address=45.91.116.14:4192 , piece_id=3NXXBKZIKSYW7WTMXBLNFEITE34CSKW5Z6TWILMTXK5DUR5TZNUA, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.888264803 (498h42m15.759347848s) [noise=true , address=145.79.227.52:4447 , piece_id=WTWT46GXXQEPO4CFV3OFVGEUDTZUCDKPHXA3LTNHPWW4ZYKTJLBQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.888721663 (498h42m15.759804708s) [noise=true , address=145.79.227.29:4941 , piece_id=WDNQR3PB5CHIEXRT7JUGES5QFKFP2RM4AE654SLCZYTOZCVA2RUQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.889172822 (498h42m15.760255867s) [noise=true , address=77.90.31.14:4664 , piece_id=U44PHPQSALZMQGSA6Z5DL47SGETV7QRF6ZAYXL5WOJIGCCW3TPNQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.889632547 (498h42m15.760715592s) [noise=true , address=145.79.227.36:4518 , piece_id=KBBEXZKJIXGCND5HKB3K67ZSCUPPUJ7RG7NVM6YCJKRBXS5WOCJA, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.890034543 (498h42m15.761117588s) [noise=true , address=145.79.227.19:4077 , piece_id=BYGCFOQY7DCXKH37EA7HDJ536RTCECGT4MY3DMVPIOIX2M47EWQQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.891411825 (498h42m15.76249486s) [noise=true , address=145.79.227.42:4123 , piece_id=6OYLBUK7AOC4UPBC7ODNV5EIUZZRB3FV7S35XYHRBDW3YACQX75A, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.891461630 (498h42m15.762544675s) [noise=true , address=145.79.227.47:4267 , piece_id=EZFWVH6EHMPPGYLE7UIYOGAUZKK5Z36EIRRJLMU2YIQMFNTTFZCQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.892737720 (498h42m15.763820755s) [noise=true , address=145.79.227.39:4014 , piece_id=3OKLQ5L2NNJNACGB7NDF5EAFKUF7XDK7W2FMVHOBP3EM66ALS3ZA, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.892912352 (498h42m15.763995387s) [noise=true , address=145.79.227.45:4231 , piece_id=CJZKVYM2RPLY3UO76YGCCCZ7X56VZRSWYE7BYEQKSI7UQBGAY22A, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.893928177 (498h42m15.765011212s) [noise=true , address=145.79.227.17:4970 , piece_id=FE66GXX3THEF4GIA4WFZMGY252M6S67TIQWIIXNOEPI5SWDDPVHQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.894298582 (498h42m15.765381617s) [noise=true , address=45.91.116.10:4083 , piece_id=P6GPNBTQFIH4TYCZ6BWEAA5IPBGGXZOVXNZFPU3WC7B7JPKH6LHA, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.895205158 (498h42m15.766288203s) [noise=true , address=77.90.31.12:4590 , piece_id=T4FDMORC6W4DEZ4KEZBU47ZM35OUBYXOXVFTO7HDWJCVWDNYLMEA, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.896299492 (498h42m15.767382538s) [noise=true , address=145.79.227.43:4158 , piece_id=DBZFUKSEQIQ3ZQWR4WCXMFSDLC5BOHFKOKHVKKNLX45GZJ2CCWJQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:17.515245304 (498h42m17.386328339s) [upload=437717]: Next returned nil segment Dec 26 10:11:17.515261535 (498h42m17.38634457s) [upload=437717]: Waiting for error group managing segments... Dec 26 10:11:17.902169444 (498h42m17.773252489s) [noise=true , address=145.79.227.47:4267 , piece_id=EZFWVH6EHMPPGYLE7UIYOGAUZKK5Z36EIRRJLMU2YIQMFNTTFZCQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.902542795 (498h42m17.77362584s) [noise=true , address=145.79.227.52:4447 , piece_id=WTWT46GXXQEPO4CFV3OFVGEUDTZUCDKPHXA3LTNHPWW4ZYKTJLBQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.910143398 (498h42m17.781226433s) [noise=true , address=145.79.227.17:4970 , piece_id=FE66GXX3THEF4GIA4WFZMGY252M6S67TIQWIIXNOEPI5SWDDPVHQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.916181516 (498h42m17.787264551s) [noise=true , address=145.79.227.19:4077 , piece_id=BYGCFOQY7DCXKH37EA7HDJ536RTCECGT4MY3DMVPIOIX2M47EWQQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.917555872 (498h42m17.788638907s) [noise=true , address=145.79.227.39:4014 , piece_id=3OKLQ5L2NNJNACGB7NDF5EAFKUF7XDK7W2FMVHOBP3EM66ALS3ZA, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.921961953 (498h42m17.793044988s) [noise=true , address=145.79.227.45:4231 , piece_id=CJZKVYM2RPLY3UO76YGCCCZ7X56VZRSWYE7BYEQKSI7UQBGAY22A, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.927337478 (498h42m17.798420513s) [noise=true , address=145.79.227.42:4123 , piece_id=6OYLBUK7AOC4UPBC7ODNV5EIUZZRB3FV7S35XYHRBDW3YACQX75A, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.927584038 (498h42m17.798667083s) [noise=true , address=145.79.227.29:4941 , piece_id=WDNQR3PB5CHIEXRT7JUGES5QFKFP2RM4AE654SLCZYTOZCVA2RUQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.931559068 (498h42m17.802642103s) [noise=true , address=145.79.227.43:4158 , piece_id=DBZFUKSEQIQ3ZQWR4WCXMFSDLC5BOHFKOKHVKKNLX45GZJ2CCWJQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.933044495 (498h42m17.804127531s) [noise=true , address=145.79.227.36:4518 , piece_id=KBBEXZKJIXGCND5HKB3K67ZSCUPPUJ7RG7NVM6YCJKRBXS5WOCJA, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:18.274353196 (498h42m18.145436232s) [noise=true , address=77.90.31.14:4664 , piece_id=U44PHPQSALZMQGSA6Z5DL47SGETV7QRF6ZAYXL5WOJIGCCW3TPNQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:18.280321731 (498h42m18.151404766s) [noise=true , address=45.91.116.14:4192 , piece_id=3NXXBKZIKSYW7WTMXBLNFEITE34CSKW5Z6TWILMTXK5DUR5TZNUA, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:18.303182202 (498h42m18.174265247s) [noise=true , address=45.91.116.10:4083 , piece_id=P6GPNBTQFIH4TYCZ6BWEAA5IPBGGXZOVXNZFPU3WC7B7JPKH6LHA, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:18.303292652 (498h42m18.174375697s) [seg_pos={0 0}, upload=437717]: Segment reached optimal threshold of 13 pieces. Dec 26 10:11:18.303488155 (498h42m18.17457119s) [noise=true , address=77.90.31.16:4735 , piece_id=VOZWPS377M4XODPZQY4KJKJGBDDRMBSXV6DM627MCQ5OF3IC25FQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 10:11:18.303497442 (498h42m18.174580477s) [noise=true , address=77.90.31.16:4735 , piece_id=VOZWPS377M4XODPZQY4KJKJGBDDRMBSXV6DM627MCQ5OF3IC25FQ, seg_pos={0 0}, upload=437717]: Long tail cancellation due to optimal threshold Dec 26 10:11:18.303658068 (498h42m18.174741113s) [noise=true , address=45.91.116.7:4009 , piece_id=UPXB3KAIFMWIK72GLQ3QY7QG7K2D3OPH2L3Z4OCDXCYQHHSHH67A, seg_pos={0 0}, upload=437717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 10:11:18.303680891 (498h42m18.174763926s) [noise=true , address=45.91.116.7:4009 , piece_id=UPXB3KAIFMWIK72GLQ3QY7QG7K2D3OPH2L3Z4OCDXCYQHHSHH67A, seg_pos={0 0}, upload=437717]: Long tail cancellation due to optimal threshold Dec 26 10:11:18.303720657 (498h42m18.174803772s) [noise=true , address=77.90.31.12:4590 , piece_id=T4FDMORC6W4DEZ4KEZBU47ZM35OUBYXOXVFTO7HDWJCVWDNYLMEA, seg_pos={0 0}, upload=437717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 10:11:18.303731578 (498h42m18.174814613s) [noise=true , address=77.90.31.12:4590 , piece_id=T4FDMORC6W4DEZ4KEZBU47ZM35OUBYXOXVFTO7HDWJCVWDNYLMEA, seg_pos={0 0}, upload=437717]: Long tail cancellation due to optimal threshold Dec 26 10:11:18.303755824 (498h42m18.174838869s) [seg_pos={0 0}, upload=437717]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:11:18.303902974 (498h42m18.17498602s) [upload=437717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:11:18.303911911 (498h42m18.174994946s) [upload=437717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:11:18.594830410 (498h42m18.465913455s) [upload=437717]: Done uploading segments... Dec 26 10:11:18.594857051 (498h42m18.465940086s) [upload=437717]: Upload finished. err: <nil> Duration: 6.161438091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X......X.........X..................X.] duration: 48.022985271s trace: https://jaeger.dev.storj.io/trace/3eea9e0464bed0e7 FAIL (2025-12-26 10:11:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.333867109 (498h40m58.204950154s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.334611487 (498h40m58.205694542s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.335747089 (498h40m58.206830124s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336515553 (498h40m58.207598598s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336870569 (498h40m58.207953614s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337310336 (498h40m58.208393381s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337723112 (498h40m58.208806157s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338318425 (498h40m58.20940147s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338828727 (498h40m58.209911772s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339185056 (498h40m58.210268091s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339640863 (498h40m58.210723909s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339998325 (498h40m58.21108136s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.340445936 (498h40m58.211528981s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.341374635 (498h40m58.21245768s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:59.798768304 (498h40m59.669851359s) [upload=437697]: Next returned nil segment Dec 26 10:09:59.798783604 (498h40m59.669866649s) [upload=437697]: Waiting for error group managing segments... Dec 26 10:10:00.202844345 (498h41m0.07392739s) [noise=true , address=145.79.227.43:4175 , piece_id=5AW772QNP72OEKJTB6I4JGD4L5ONMRF5ZNXYLMKZAQDC56VOLSLQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.212993169 (498h41m0.084076204s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.224811985 (498h41m0.09589503s) [noise=true , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.239828058 (498h41m0.110911093s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.411596484 (498h41m0.282679529s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.490073348 (498h41m0.361156383s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.496009011 (498h41m0.367092056s) [noise=true , address=77.90.31.21:4904 , piece_id=NUBWDV2G25QBQKJDLQWVQG5JIE7PUW5XRHYNRCTVM3EENZPNQ6YA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.505536983 (498h41m0.376620018s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.621872607 (498h41m0.492955642s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.628231495 (498h41m0.49931453s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.634412284 (498h41m0.505495329s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.069090817 (498h41m0.940173852s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178406242 (498h41m1.049489277s) [noise=true , address=145.79.227.44:4226 , piece_id=CWN7Z7L2QIYNO75WNLF6Q4PPM3EIEAYJGYJCCY47U6NT4E4PYEVQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178763302 (498h41m1.049846347s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178785645 (498h41m1.04986869s) [seg_pos={0 0}, upload=437697]: Segment reached optimal threshold of 14 pieces. Dec 26 10:10:01.179093852 (498h41m1.050176897s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection Dec 26 10:10:01.179102047 (498h41m1.050185092s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179217547 (498h41m1.050300592s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection Dec 26 10:10:01.179223299 (498h41m1.050306334s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179319030 (498h41m1.050402075s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AZxy9wnMgVu2PdFNtFrShYhWfF6PL4mgWumjqVk68voKsnMk6): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179328499 (498h41m1.050411544s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179462273 (498h41m1.050545308s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179468526 (498h41m1.050551561s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179504985 (498h41m1.05058803s) [seg_pos={0 0}, upload=437697]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:10:01.179608212 (498h41m1.050691267s) [upload=437697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:10:01.179618601 (498h41m1.050701656s) [upload=437697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:10:01.469507960 (498h41m1.340591005s) [upload=437697]: Done uploading segments... Dec 26 10:10:01.469529561 (498h41m1.340612606s) [upload=437697]: Upload finished. err: <nil> Duration: 48.023002996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [..X.....X.X...X..XXXXXXX.....XX.XXX.X..X.XXXXX.X.X..XX...X.X] duration: 30.001696571s trace: https://jaeger.dev.storj.io/trace/97d1e0d9e37debb FAIL (2025-12-26 10:11:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 10:11:15.640525945 (498h42m15.51160899s) [upload=437717]: Starting upload Dec 26 10:11:15.640625635 (498h42m15.51170867s) [upload=437717]: Done starting upload Dec 26 10:11:15.640690639 (498h42m15.511773674s) [upload=437717]: Uploading segments... Dec 26 10:11:15.640875571 (498h42m15.511958616s) [upload=437717]: Got next segment. Inline: false Dec 26 10:11:15.640913042 (498h42m15.511996077s) [upload=437717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:11:15.640919003 (498h42m15.512002038s) [upload=437717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:11:15.887069547 (498h42m15.758152582s) [seg_pos={0 0}, upload=437717]: Begin upload segment... Dec 26 10:11:15.887113841 (498h42m15.758196876s) [seg_pos={0 0}, upload=437717]: Done begin upload segment. Dec 26 10:11:15.887139410 (498h42m15.758222455s) [noise=true , address=45.91.116.7:4009 , piece_id=UPXB3KAIFMWIK72GLQ3QY7QG7K2D3OPH2L3Z4OCDXCYQHHSHH67A, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.887183634 (498h42m15.758266679s) [noise=true , address=77.90.31.16:4735 , piece_id=VOZWPS377M4XODPZQY4KJKJGBDDRMBSXV6DM627MCQ5OF3IC25FQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.887780100 (498h42m15.758863135s) [noise=true , address=45.91.116.14:4192 , piece_id=3NXXBKZIKSYW7WTMXBLNFEITE34CSKW5Z6TWILMTXK5DUR5TZNUA, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.888264803 (498h42m15.759347848s) [noise=true , address=145.79.227.52:4447 , piece_id=WTWT46GXXQEPO4CFV3OFVGEUDTZUCDKPHXA3LTNHPWW4ZYKTJLBQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.888721663 (498h42m15.759804708s) [noise=true , address=145.79.227.29:4941 , piece_id=WDNQR3PB5CHIEXRT7JUGES5QFKFP2RM4AE654SLCZYTOZCVA2RUQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.889172822 (498h42m15.760255867s) [noise=true , address=77.90.31.14:4664 , piece_id=U44PHPQSALZMQGSA6Z5DL47SGETV7QRF6ZAYXL5WOJIGCCW3TPNQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.889632547 (498h42m15.760715592s) [noise=true , address=145.79.227.36:4518 , piece_id=KBBEXZKJIXGCND5HKB3K67ZSCUPPUJ7RG7NVM6YCJKRBXS5WOCJA, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.890034543 (498h42m15.761117588s) [noise=true , address=145.79.227.19:4077 , piece_id=BYGCFOQY7DCXKH37EA7HDJ536RTCECGT4MY3DMVPIOIX2M47EWQQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.891411825 (498h42m15.76249486s) [noise=true , address=145.79.227.42:4123 , piece_id=6OYLBUK7AOC4UPBC7ODNV5EIUZZRB3FV7S35XYHRBDW3YACQX75A, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.891461630 (498h42m15.762544675s) [noise=true , address=145.79.227.47:4267 , piece_id=EZFWVH6EHMPPGYLE7UIYOGAUZKK5Z36EIRRJLMU2YIQMFNTTFZCQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.892737720 (498h42m15.763820755s) [noise=true , address=145.79.227.39:4014 , piece_id=3OKLQ5L2NNJNACGB7NDF5EAFKUF7XDK7W2FMVHOBP3EM66ALS3ZA, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.892912352 (498h42m15.763995387s) [noise=true , address=145.79.227.45:4231 , piece_id=CJZKVYM2RPLY3UO76YGCCCZ7X56VZRSWYE7BYEQKSI7UQBGAY22A, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.893928177 (498h42m15.765011212s) [noise=true , address=145.79.227.17:4970 , piece_id=FE66GXX3THEF4GIA4WFZMGY252M6S67TIQWIIXNOEPI5SWDDPVHQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.894298582 (498h42m15.765381617s) [noise=true , address=45.91.116.10:4083 , piece_id=P6GPNBTQFIH4TYCZ6BWEAA5IPBGGXZOVXNZFPU3WC7B7JPKH6LHA, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.895205158 (498h42m15.766288203s) [noise=true , address=77.90.31.12:4590 , piece_id=T4FDMORC6W4DEZ4KEZBU47ZM35OUBYXOXVFTO7HDWJCVWDNYLMEA, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:15.896299492 (498h42m15.767382538s) [noise=true , address=145.79.227.43:4158 , piece_id=DBZFUKSEQIQ3ZQWR4WCXMFSDLC5BOHFKOKHVKKNLX45GZJ2CCWJQ, seg_pos={0 0}, upload=437717]: Uploading piece... Dec 26 10:11:17.515245304 (498h42m17.386328339s) [upload=437717]: Next returned nil segment Dec 26 10:11:17.515261535 (498h42m17.38634457s) [upload=437717]: Waiting for error group managing segments... Dec 26 10:11:17.902169444 (498h42m17.773252489s) [noise=true , address=145.79.227.47:4267 , piece_id=EZFWVH6EHMPPGYLE7UIYOGAUZKK5Z36EIRRJLMU2YIQMFNTTFZCQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.902542795 (498h42m17.77362584s) [noise=true , address=145.79.227.52:4447 , piece_id=WTWT46GXXQEPO4CFV3OFVGEUDTZUCDKPHXA3LTNHPWW4ZYKTJLBQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.910143398 (498h42m17.781226433s) [noise=true , address=145.79.227.17:4970 , piece_id=FE66GXX3THEF4GIA4WFZMGY252M6S67TIQWIIXNOEPI5SWDDPVHQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.916181516 (498h42m17.787264551s) [noise=true , address=145.79.227.19:4077 , piece_id=BYGCFOQY7DCXKH37EA7HDJ536RTCECGT4MY3DMVPIOIX2M47EWQQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.917555872 (498h42m17.788638907s) [noise=true , address=145.79.227.39:4014 , piece_id=3OKLQ5L2NNJNACGB7NDF5EAFKUF7XDK7W2FMVHOBP3EM66ALS3ZA, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.921961953 (498h42m17.793044988s) [noise=true , address=145.79.227.45:4231 , piece_id=CJZKVYM2RPLY3UO76YGCCCZ7X56VZRSWYE7BYEQKSI7UQBGAY22A, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.927337478 (498h42m17.798420513s) [noise=true , address=145.79.227.42:4123 , piece_id=6OYLBUK7AOC4UPBC7ODNV5EIUZZRB3FV7S35XYHRBDW3YACQX75A, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.927584038 (498h42m17.798667083s) [noise=true , address=145.79.227.29:4941 , piece_id=WDNQR3PB5CHIEXRT7JUGES5QFKFP2RM4AE654SLCZYTOZCVA2RUQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.931559068 (498h42m17.802642103s) [noise=true , address=145.79.227.43:4158 , piece_id=DBZFUKSEQIQ3ZQWR4WCXMFSDLC5BOHFKOKHVKKNLX45GZJ2CCWJQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:17.933044495 (498h42m17.804127531s) [noise=true , address=145.79.227.36:4518 , piece_id=KBBEXZKJIXGCND5HKB3K67ZSCUPPUJ7RG7NVM6YCJKRBXS5WOCJA, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:18.274353196 (498h42m18.145436232s) [noise=true , address=77.90.31.14:4664 , piece_id=U44PHPQSALZMQGSA6Z5DL47SGETV7QRF6ZAYXL5WOJIGCCW3TPNQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:18.280321731 (498h42m18.151404766s) [noise=true , address=45.91.116.14:4192 , piece_id=3NXXBKZIKSYW7WTMXBLNFEITE34CSKW5Z6TWILMTXK5DUR5TZNUA, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:18.303182202 (498h42m18.174265247s) [noise=true , address=45.91.116.10:4083 , piece_id=P6GPNBTQFIH4TYCZ6BWEAA5IPBGGXZOVXNZFPU3WC7B7JPKH6LHA, seg_pos={0 0}, upload=437717]: Done uploading piece. err: <nil> Dec 26 10:11:18.303292652 (498h42m18.174375697s) [seg_pos={0 0}, upload=437717]: Segment reached optimal threshold of 13 pieces. Dec 26 10:11:18.303488155 (498h42m18.17457119s) [noise=true , address=77.90.31.16:4735 , piece_id=VOZWPS377M4XODPZQY4KJKJGBDDRMBSXV6DM627MCQ5OF3IC25FQ, seg_pos={0 0}, upload=437717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 10:11:18.303497442 (498h42m18.174580477s) [noise=true , address=77.90.31.16:4735 , piece_id=VOZWPS377M4XODPZQY4KJKJGBDDRMBSXV6DM627MCQ5OF3IC25FQ, seg_pos={0 0}, upload=437717]: Long tail cancellation due to optimal threshold Dec 26 10:11:18.303658068 (498h42m18.174741113s) [noise=true , address=45.91.116.7:4009 , piece_id=UPXB3KAIFMWIK72GLQ3QY7QG7K2D3OPH2L3Z4OCDXCYQHHSHH67A, seg_pos={0 0}, upload=437717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 10:11:18.303680891 (498h42m18.174763926s) [noise=true , address=45.91.116.7:4009 , piece_id=UPXB3KAIFMWIK72GLQ3QY7QG7K2D3OPH2L3Z4OCDXCYQHHSHH67A, seg_pos={0 0}, upload=437717]: Long tail cancellation due to optimal threshold Dec 26 10:11:18.303720657 (498h42m18.174803772s) [noise=true , address=77.90.31.12:4590 , piece_id=T4FDMORC6W4DEZ4KEZBU47ZM35OUBYXOXVFTO7HDWJCVWDNYLMEA, seg_pos={0 0}, upload=437717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 26 10:11:18.303731578 (498h42m18.174814613s) [noise=true , address=77.90.31.12:4590 , piece_id=T4FDMORC6W4DEZ4KEZBU47ZM35OUBYXOXVFTO7HDWJCVWDNYLMEA, seg_pos={0 0}, upload=437717]: Long tail cancellation due to optimal threshold Dec 26 10:11:18.303755824 (498h42m18.174838869s) [seg_pos={0 0}, upload=437717]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:11:18.303902974 (498h42m18.17498602s) [upload=437717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:11:18.303911911 (498h42m18.174994946s) [upload=437717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:11:18.594830410 (498h42m18.465913455s) [upload=437717]: Done uploading segments... Dec 26 10:11:18.594857051 (498h42m18.465940086s) [upload=437717]: Upload finished. err: <nil> Duration: 30.001725525s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X......X.........X..................X] duration: 1m0.001780394s trace: https://jaeger.dev.storj.io/trace/3eea9e0464bed0e7 FAIL (2025-12-26 10:10:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.333867109 (498h40m58.204950154s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.334611487 (498h40m58.205694542s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.335747089 (498h40m58.206830124s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336515553 (498h40m58.207598598s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.336870569 (498h40m58.207953614s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337310336 (498h40m58.208393381s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.337723112 (498h40m58.208806157s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338318425 (498h40m58.20940147s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.338828727 (498h40m58.209911772s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339185056 (498h40m58.210268091s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339640863 (498h40m58.210723909s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.339998325 (498h40m58.21108136s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.340445936 (498h40m58.211528981s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:58.341374635 (498h40m58.21245768s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Uploading piece... Dec 26 10:09:59.798768304 (498h40m59.669851359s) [upload=437697]: Next returned nil segment Dec 26 10:09:59.798783604 (498h40m59.669866649s) [upload=437697]: Waiting for error group managing segments... Dec 26 10:10:00.202844345 (498h41m0.07392739s) [noise=true , address=145.79.227.43:4175 , piece_id=5AW772QNP72OEKJTB6I4JGD4L5ONMRF5ZNXYLMKZAQDC56VOLSLQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.212993169 (498h41m0.084076204s) [noise=true , address=145.79.227.52:4448 , piece_id=IWM23PIYPHBPVBHYVI4NUX3ONLI4TRV72RA3SWRYDVBLJWEMEEWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.224811985 (498h41m0.09589503s) [noise=true , address=145.79.227.46:4628 , piece_id=XNBXOGZZLRWLL3345IN2RLM5WFZIJO7CA6ZJIV3GEMDH2TR2CBQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.239828058 (498h41m0.110911093s) [noise=true , address=145.79.227.23:4580 , piece_id=2EVXDPS3DQD3HMM6KYNSJIM6CCFL4MHOOTVWOVVGLNGV2YUPRTUA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.411596484 (498h41m0.282679529s) [noise=true , address=77.90.31.16:4756 , piece_id=6OSMLJ7TNZSDKH6HIT5HUPPGBH7NHNP6D77EAXNP4THGUCUJXBWQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.490073348 (498h41m0.361156383s) [noise=true , address=77.90.31.11:4577 , piece_id=XJBUNBYBGWKIB4IO6CKLXI57JC72T7JPKLVPXI5O264TQGFJPL6A, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.496009011 (498h41m0.367092056s) [noise=true , address=77.90.31.21:4904 , piece_id=NUBWDV2G25QBQKJDLQWVQG5JIE7PUW5XRHYNRCTVM3EENZPNQ6YA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.505536983 (498h41m0.376620018s) [noise=true , address=77.90.31.15:4722 , piece_id=HR7YBJC6WUKS5TRZ6AWMGD5KHOVH4WMMW5D745TY25ZCBZYCWLMA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.621872607 (498h41m0.492955642s) [noise=true , address=45.91.116.4:4963 , piece_id=LO4HAJNT7O36V4ON6PPLW7JVFMPZ7OFNJGUAVM4KZY43CXZRUE2Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.628231495 (498h41m0.49931453s) [noise=true , address=77.90.31.12:4599 , piece_id=N2QJGIN3L3TF7AIUOP4VTLUARVTTRZHHP4QOD6DYNGEYN34OWAQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:00.634412284 (498h41m0.505495329s) [noise=true , address=77.90.31.19:4874 , piece_id=A2GJ54F3PNAX7LHM6JLKSPLCLIJSIPWDIDYBBDWL3YMTNGNCCKZA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.069090817 (498h41m0.940173852s) [noise=true , address=145.79.227.30:4791 , piece_id=EJCYOE4UHASPUFQ6B2CTEAV5GMSJCSEVYWZJEZXZBG4VIHXCFJQQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178406242 (498h41m1.049489277s) [noise=true , address=145.79.227.44:4226 , piece_id=CWN7Z7L2QIYNO75WNLF6Q4PPM3EIEAYJGYJCCY47U6NT4E4PYEVQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178763302 (498h41m1.049846347s) [noise=true , address=45.91.116.7:4042 , piece_id=WGJ55EX2HMJ26AMPIW2AP4BBX2NGH3TGMMJREVOIPL4YYC27JLSQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: <nil> Dec 26 10:10:01.178785645 (498h41m1.04986869s) [seg_pos={0 0}, upload=437697]: Segment reached optimal threshold of 14 pieces. Dec 26 10:10:01.179093852 (498h41m1.050176897s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59788->145.79.227.33:4901: use of closed network connection Dec 26 10:10:01.179102047 (498h41m1.050185092s) [noise=true , address=145.79.227.33:4901 , piece_id=UDFTT7JKXRROICHN5JTIM4KPWPRIUYMFAIBAZP6WYKTO633Z4BKQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179217547 (498h41m1.050300592s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44476->64.49.13.169:4709: use of closed network connection Dec 26 10:10:01.179223299 (498h41m1.050306334s) [noise=true , address=64.49.13.169:4709 , piece_id=DUSNIXDOFE7JRLV2VXQJOKKV4G2SOAFSLUE73RNIDWYFYYXTJS6Q, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179319030 (498h41m1.050402075s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AZxy9wnMgVu2PdFNtFrShYhWfF6PL4mgWumjqVk68voKsnMk6): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179328499 (498h41m1.050411544s) [noise=true , address=77.90.31.18:4811 , piece_id=4M34NK6LOW3K7CYCOVW4BATJGOZNDBH62KFOHIPU7M6OKA5VJLNQ, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179462273 (498h41m1.050545308s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7): protocol: expected piece hash; context canceled; EOF Dec 26 10:10:01.179468526 (498h41m1.050551561s) [noise=true , address=77.90.31.10:4548 , piece_id=VI7FGGUZ7MXT674LEDJTEYL4DA37XCIYJBT2PHR3BGXXFHUKJSOA, seg_pos={0 0}, upload=437697]: Long tail cancellation due to optimal threshold Dec 26 10:10:01.179504985 (498h41m1.05058803s) [seg_pos={0 0}, upload=437697]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:10:01.179608212 (498h41m1.050691267s) [upload=437697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:10:01.179618601 (498h41m1.050701656s) [upload=437697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:10:01.469507960 (498h41m1.340591005s) [upload=437697]: Done uploading segments... Dec 26 10:10:01.469529561 (498h41m1.340612606s) [upload=437697]: Upload finished. err: <nil> Duration: 1m0.001790854s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)