Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX...............X..] duration: 10.89577914s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 20:56:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 10.895807935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..............X.....................................X...] duration: 9.466243249s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 20:56:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 9.466267725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..............X.....................................X..] duration: 15.294929525s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 20:55:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 15.294955754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX...............X.] duration: 22.576799698s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 20:55:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 22.576820648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..............X.....................................X.] duration: 6.66081701s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 20:55:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 6.660833421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..............X.....................................X] duration: 1m0.003667736s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 20:55:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 1m0.00369067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX...............X] duration: 30.008237729s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 20:55:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 30.008265202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX...............] duration: 6.062260802s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:53:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 6.062294415s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX..............] duration: 15.210592812s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:53:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 15.210621436s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..............X.....................................] duration: 8.537361854s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:53:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 8.53738632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX.............] duration: 7.881349148s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:53:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 7.881378155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX............] duration: 20.261897185s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:52:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 20.261923613s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..............X....................................] duration: 14.759617947s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:52:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 14.759646161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..............X...................................] duration: 7.039120786s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:52:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 7.039156304s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX...........] duration: 6.492951133s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:51:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 6.492978364s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..............X..................................] duration: 10.601776978s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:51:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 10.601801714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..............X.................................] duration: 15.391686473s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:51:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 15.391714978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX..........] duration: 6.3184914s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:51:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 6.318518683s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..............X................................] duration: 17.988784584s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:50:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 17.988809059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X..............X...............................] duration: 7.132216229s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:50:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 7.132254229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX.........] duration: 6.200946857s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:50:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 6.200978016s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.....XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX........] duration: 7.945398452s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:49:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 7.945426856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X..............X..............................] duration: 25.996833221s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:49:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 25.996877506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X..............X.............................] duration: 5.950060283s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:48:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 5.950075251s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [......XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX.......] duration: 6.081986414s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:48:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 6.08200578s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X..............X............................] duration: 24.044554577s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:48:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 24.044576468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.......XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX......] duration: 8.880183509s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:48:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 8.880207595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX.....] duration: 26.336873741s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:48:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 26.336904498s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X..............X...........................] duration: 36.932073938s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:47:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 36.932112871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX....] duration: 13.14927086s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:47:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 13.149300767s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X..............X..........................] duration: 7.479382433s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:46:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 7.479408301s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X..............X.........................] duration: 22.604321948s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:46:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 22.604341716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX...] duration: 6.01106698s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:46:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 6.01109358s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX..] duration: 10.858690919s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:45:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 10.858762625s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X..............X........................] duration: 13.843051043s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:45:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 13.843080349s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX.] duration: 27.383321548s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:45:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 27.383364249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX] duration: 30.002690585s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:44:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 30.002703109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..............X.......................] duration: 16.372425717s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:44:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 16.372453049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XX] duration: 30.002963659s trace: https://jaeger.dev.storj.io/trace/6a63853ece39ac5d FAIL (2025-12-26 20:44:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: OJKLHZP4QUL6WC4CUBC4ALNNTS6BNLZAKDEIIUPZU4HNTUAMUYLQ) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 4TDWDS6V3R5QA2PZ6AZRTKWLVKRASW6Y5KU7KCKULSTUYJD5C26A) context deadline exceeded ======LOGS====== 26 20:43:31.827779558 (509h14m31.698862594s) [upload=446859]: Starting upload Dec 26 20:43:31.827834593 (509h14m31.698917628s) [upload=446859]: Done starting upload Dec 26 20:43:31.828014786 (509h14m31.699097821s) [upload=446859]: Uploading segments... Dec 26 20:43:31.828088216 (509h14m31.699171261s) [upload=446859]: Got next segment. Inline: false Dec 26 20:43:31.828110468 (509h14m31.699193503s) [upload=446859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:43:31.828114536 (509h14m31.699197581s) [upload=446859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:43:32.073213258 (509h14m31.944296293s) [seg_pos={0 0}, upload=446859]: Begin upload segment... Dec 26 20:43:32.073256430 (509h14m31.944339475s) [seg_pos={0 0}, upload=446859]: Done begin upload segment. Dec 26 20:43:32.073284083 (509h14m31.944367118s) [noise=true , address=145.79.227.46:4602 , piece_id=SJ6QBMVV26ICJIMHIZLCSIAXLRD45XSMOL3JAPCA4XWYMG4E3S7A, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.073298760 (509h14m31.944381795s) [noise=true , address=45.91.116.12:4228 , piece_id=4TDWDS6V3R5QA2PZ6AZRTKWLVKRASW6Y5KU7KCKULSTUYJD5C26A, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.073853016 (509h14m31.944936061s) [noise=true , address=145.79.227.36:4518 , piece_id=BMTNDHPAGDUHOMT5II55URT6YNLGHDIJTWDJZNLQ6OMBOD6VXM2A, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.074085579 (509h14m31.945168614s) [noise=true , address=145.79.227.15:4874 , piece_id=ZC3YBACI5X6LSMLVSYDHP2JLGKJQY25FLUDWNF4BNQEAZRVKHLGA, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.074954754 (509h14m31.946037789s) [noise=true , address=145.79.227.25:4618 , piece_id=25GETOMMHIJVJNDMMMXDGICN3KAD5UEGE2HQZS5V423ZLOXF735A, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.075343073 (509h14m31.946426119s) [noise=true , address=145.79.227.17:4972 , piece_id=H3JJJ2IZHMLPCUKOBJWCWG4AF7FNWV7AJ3ZUKTDL5SGLDHYAMD3A, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.075608128 (509h14m31.946691163s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZDVAHMG4KAP5TJZGC35WM2RXBADDZMJNBQ7AK7BH3LCKMWZ62FA, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.075904713 (509h14m31.946987758s) [noise=true , address=45.91.116.11:4118 , piece_id=SY7SPKVSDSMHJVR2QXA3OGLN2QAVQS2MNDKJIQRLBO5I75M3WQ3A, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.076406608 (509h14m31.947489653s) [noise=true , address=77.90.31.16:4735 , piece_id=BYV2EM7YXVYAU5QNUFD7MY6MFJV6UVVWG5S7YEZBOWHDOJR7KPPQ, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.076415315 (509h14m31.94749835s) [noise=true , address=45.91.116.6:4976 , piece_id=KK3H5QXVJM32VWJZ2UFGKJ6Q54OPCWGHY7JTCUHBD2EUKJLXVF5Q, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.076456674 (509h14m31.947539709s) [noise=true , address=45.91.116.10:4083 , piece_id=UQZ3HBON554YMGZQ7QCWQYGHFOCHSCBFPHGVM7GAMCWPOS23Z3RA, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.076750192 (509h14m31.947833227s) [noise=true , address=45.91.116.4:4939 , piece_id=PGIEUHTDQUW7UPFLQRNKYJS6VKOYU2LLLGLSGXFRYP36YVS47HHA, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.076853990 (509h14m31.947937035s) [noise=true , address=77.90.31.10:4520 , piece_id=PE54A53EAKOZXPXVIO64IWKOMYIB5QGRSMCS4NGEGPBF74UOGKZQ, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.077033963 (509h14m31.948117008s) [noise=true , address=145.79.227.22:4509 , piece_id=JWZNBWRS23HVDOLTA3XCJYTUU6UMRCF6TCI43J7HISLLYD4YXI2A, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.077428595 (509h14m31.94851164s) [noise=true , address=45.91.116.5:4297 , piece_id=OJKLHZP4QUL6WC4CUBC4ALNNTS6BNLZAKDEIIUPZU4HNTUAMUYLQ, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:32.077697115 (509h14m31.94878015s) [noise=true , address=145.79.227.52:4447 , piece_id=NIUE3X627GJQEW4II3TGQVOVSFCQDT4Y7VEKRC3IHTINO6FFRQVA, seg_pos={0 0}, upload=446859]: Uploading piece... Dec 26 20:43:33.786420078 (509h14m33.657503123s) [upload=446859]: Next returned nil segment Dec 26 20:43:33.786458861 (509h14m33.657541897s) [upload=446859]: Waiting for error group managing segments... Dec 26 20:43:34.071567109 (509h14m33.942650144s) [noise=true , address=145.79.227.15:4874 , piece_id=ZC3YBACI5X6LSMLVSYDHP2JLGKJQY25FLUDWNF4BNQEAZRVKHLGA, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.076340007 (509h14m33.947423042s) [noise=true , address=145.79.227.46:4602 , piece_id=SJ6QBMVV26ICJIMHIZLCSIAXLRD45XSMOL3JAPCA4XWYMG4E3S7A, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.078259822 (509h14m33.949342857s) [noise=true , address=145.79.227.25:4618 , piece_id=25GETOMMHIJVJNDMMMXDGICN3KAD5UEGE2HQZS5V423ZLOXF735A, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.080296029 (509h14m33.951379074s) [noise=true , address=145.79.227.36:4518 , piece_id=BMTNDHPAGDUHOMT5II55URT6YNLGHDIJTWDJZNLQ6OMBOD6VXM2A, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.098910434 (509h14m33.96999347s) [noise=true , address=145.79.227.52:4447 , piece_id=NIUE3X627GJQEW4II3TGQVOVSFCQDT4Y7VEKRC3IHTINO6FFRQVA, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.108246703 (509h14m33.979329738s) [noise=true , address=145.79.227.22:4509 , piece_id=JWZNBWRS23HVDOLTA3XCJYTUU6UMRCF6TCI43J7HISLLYD4YXI2A, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.322044521 (509h14m34.193127556s) [noise=true , address=45.91.116.5:4297 , piece_id=OJKLHZP4QUL6WC4CUBC4ALNNTS6BNLZAKDEIIUPZU4HNTUAMUYLQ, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.353763721 (509h14m34.224846756s) [noise=true , address=45.91.116.10:4083 , piece_id=UQZ3HBON554YMGZQ7QCWQYGHFOCHSCBFPHGVM7GAMCWPOS23Z3RA, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.356366086 (509h14m34.227449121s) [noise=true , address=45.91.116.12:4228 , piece_id=4TDWDS6V3R5QA2PZ6AZRTKWLVKRASW6Y5KU7KCKULSTUYJD5C26A, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.385228866 (509h14m34.256311911s) [noise=true , address=45.91.116.4:4939 , piece_id=PGIEUHTDQUW7UPFLQRNKYJS6VKOYU2LLLGLSGXFRYP36YVS47HHA, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.459915914 (509h14m34.330998959s) [noise=true , address=145.79.227.17:4972 , piece_id=H3JJJ2IZHMLPCUKOBJWCWG4AF7FNWV7AJ3ZUKTDL5SGLDHYAMD3A, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:34.932132016 (509h14m34.803215062s) [noise=true , address=77.90.31.10:4520 , piece_id=PE54A53EAKOZXPXVIO64IWKOMYIB5QGRSMCS4NGEGPBF74UOGKZQ, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:35.060598159 (509h14m34.931681194s) [noise=true , address=45.91.116.11:4118 , piece_id=SY7SPKVSDSMHJVR2QXA3OGLN2QAVQS2MNDKJIQRLBO5I75M3WQ3A, seg_pos={0 0}, upload=446859]: Done uploading piece. err: <nil> Dec 26 20:43:35.060617015 (509h14m34.93170005s) [seg_pos={0 0}, upload=446859]: Segment reached optimal threshold of 13 pieces. Dec 26 20:43:35.060803269 (509h14m34.931886314s) [noise=true , address=77.90.31.16:4735 , piece_id=BYV2EM7YXVYAU5QNUFD7MY6MFJV6UVVWG5S7YEZBOWHDOJR7KPPQ, seg_pos={0 0}, upload=446859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 20:43:35.060809302 (509h14m34.931892337s) [noise=true , address=77.90.31.16:4735 , piece_id=BYV2EM7YXVYAU5QNUFD7MY6MFJV6UVVWG5S7YEZBOWHDOJR7KPPQ, seg_pos={0 0}, upload=446859]: Long tail cancellation due to optimal threshold Dec 26 20:43:35.060879435 (509h14m34.93196248s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZDVAHMG4KAP5TJZGC35WM2RXBADDZMJNBQ7AK7BH3LCKMWZ62FA, seg_pos={0 0}, upload=446859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:33870->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:33870->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33870->145.79.227.5:4222: use of closed network connection Dec 26 20:43:35.060892069 (509h14m34.931975104s) [noise=true , address=145.79.227.5:4222 , piece_id=4ZDVAHMG4KAP5TJZGC35WM2RXBADDZMJNBQ7AK7BH3LCKMWZ62FA, seg_pos={0 0}, upload=446859]: Long tail cancellation due to optimal threshold Dec 26 20:43:35.061057524 (509h14m34.932140559s) [noise=true , address=45.91.116.6:4976 , piece_id=KK3H5QXVJM32VWJZ2UFGKJ6Q54OPCWGHY7JTCUHBD2EUKJLXVF5Q, seg_pos={0 0}, upload=446859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 20:43:35.061065149 (509h14m34.932148184s) [noise=true , address=45.91.116.6:4976 , piece_id=KK3H5QXVJM32VWJZ2UFGKJ6Q54OPCWGHY7JTCUHBD2EUKJLXVF5Q, seg_pos={0 0}, upload=446859]: Long tail cancellation due to optimal threshold Dec 26 20:43:35.061083473 (509h14m34.932166518s) [seg_pos={0 0}, upload=446859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:43:35.061146283 (509h14m34.932229318s) [upload=446859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:43:35.061150911 (509h14m34.932233946s) [upload=446859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:43:35.358156253 (509h14m35.229239308s) [upload=446859]: Done uploading segments... Dec 26 20:43:35.358177122 (509h14m35.229260167s) [upload=446859]: Upload finished. err: <nil> Duration: 30.002991452s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X..............X......................] duration: 22.870797964s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:43:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 22.870826841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..X] duration: 30.002601407s trace: https://jaeger.dev.storj.io/trace/76a3eb206709086d FAIL (2025-12-26 20:43:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 20:42:53.452673042 (509h13m53.323756077s) [upload=446850]: Starting upload Dec 26 20:42:53.452749768 (509h13m53.323832803s) [upload=446850]: Done starting upload Dec 26 20:42:53.452808440 (509h13m53.323891475s) [upload=446850]: Uploading segments... Dec 26 20:42:53.452948256 (509h13m53.324031301s) [upload=446850]: Got next segment. Inline: false Dec 26 20:42:53.452985026 (509h13m53.324068061s) [upload=446850]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:42:53.452989364 (509h13m53.324072399s) [upload=446850]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:42:53.699109089 (509h13m53.570192124s) [seg_pos={0 0}, upload=446850]: Begin upload segment... Dec 26 20:42:53.699154026 (509h13m53.570237061s) [seg_pos={0 0}, upload=446850]: Done begin upload segment. Dec 26 20:42:53.699180455 (509h13m53.57026349s) [noise=true , address=45.91.116.6:4974 , piece_id=P4MI4VNKCMSC56YNO52E623IXIUAKP2LNBEG7T3OP5YNKKMX7BCQ, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.699750982 (509h13m53.570834017s) [noise=true , address=145.79.227.15:4874 , piece_id=O7UKVEQ5L6Q6X4EB3KBC2WPF6US4IF5ALLAAKRTGOGO3XHQ7YD3A, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.700661506 (509h13m53.571744541s) [noise=true , address=145.79.227.48:4301 , piece_id=JX3SVJSZGJOI2RMLMUT7EVJNSXQ53BB4G5QMHEUREKNW23L4V3LQ, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.701146840 (509h13m53.572229885s) [noise=true , address=145.79.227.42:4123 , piece_id=SI2NCHXTYUN7MFEIWUDSPIUODD3O7JLCW6IZ5CPE5HVIC2EIX3LA, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.701651050 (509h13m53.572734085s) [noise=true , address=145.79.227.47:4266 , piece_id=Q4KPGM7R3MBONTHUAG5S6CGR4BDWKDEJ2ZGST6H6IEQLFJSISV2A, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.702201147 (509h13m53.573284182s) [noise=true , address=145.79.227.13:4804 , piece_id=WU7S7LN3IPTR2HFTSKLEXDMMQMYC4BDSCQKFIHKTXUH4YK3CBQTQ, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.703583289 (509h13m53.574666335s) [noise=true , address=77.90.31.10:4520 , piece_id=EVOR62CPL63GGJJQA6GXOA2WAW3PG32YXCRB77MFQNEVSDH4P24A, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.704767635 (509h13m53.57585067s) [noise=true , address=145.79.227.36:4517 , piece_id=2HDK2GDQ3V7JG3SCR235TJE6AETIWD4TCXOIWH5K23WJGTB7WABA, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.705300158 (509h13m53.576383193s) [noise=true , address=145.79.227.25:4618 , piece_id=S7VAIMY5CMJIU7PYO66XVBQGZIAYNWWIVQYEO5PCK4774A5CYBCA, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.708118705 (509h13m53.579201761s) [noise=true , address=145.79.227.38:4978 , piece_id=7IUY2X6KBIRN4LXZ7AHJKDBCLFBSO2WGDHKE7DO2NLISAY5IJGGQ, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.709352265 (509h13m53.58043531s) [noise=true , address=45.91.116.10:4083 , piece_id=VZ7HEFNEDRYHBCLYVDGROGXN7KP4274HCA5KKMKC66LFDPWMQX4A, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.712141386 (509h13m53.583224421s) [noise=true , address=145.79.227.44:4195 , piece_id=5M56H5P3H4W2PV3VFFPV33GT7TVSKHICJQ2BC5JOADZRKEIEDZDQ, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.713687509 (509h13m53.584770554s) [noise=true , address=45.91.116.12:4226 , piece_id=5RRTTAKVHXJNIW2B3EJNG654W76OKRP5SJT66UPTMYQ3RYBHNLFQ, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.715132111 (509h13m53.586215146s) [noise=true , address=145.79.227.4:4186 , piece_id=TVEWTXJF5P5HH53BHBNVUIU7EZ7LFG4XQYJEUZ46IHGOIWRBGBVA, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.716635082 (509h13m53.587718117s) [noise=true , address=145.79.227.3:4150 , piece_id=KIBZZD5MLGRMOEZMFBQODTEDF7CPDTFRN74BQ3R7WVGAOA2ZY55Q, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:53.723786299 (509h13m53.594869344s) [noise=true , address=77.90.31.14:4661 , piece_id=PUXUG2OO77TB2VWYGABS4K7NMSFGLUG77VOBZKJDGNRGSFIDT7FQ, seg_pos={0 0}, upload=446850]: Uploading piece... Dec 26 20:42:55.391698591 (509h13m55.262781626s) [upload=446850]: Next returned nil segment Dec 26 20:42:55.391712427 (509h13m55.262795462s) [upload=446850]: Waiting for error group managing segments... Dec 26 20:42:55.703427408 (509h13m55.574510453s) [noise=true , address=145.79.227.15:4874 , piece_id=O7UKVEQ5L6Q6X4EB3KBC2WPF6US4IF5ALLAAKRTGOGO3XHQ7YD3A, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.713861356 (509h13m55.584944391s) [noise=true , address=145.79.227.47:4266 , piece_id=Q4KPGM7R3MBONTHUAG5S6CGR4BDWKDEJ2ZGST6H6IEQLFJSISV2A, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.723816092 (509h13m55.594899127s) [noise=true , address=145.79.227.38:4978 , piece_id=7IUY2X6KBIRN4LXZ7AHJKDBCLFBSO2WGDHKE7DO2NLISAY5IJGGQ, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.724736855 (509h13m55.5958199s) [noise=true , address=145.79.227.48:4301 , piece_id=JX3SVJSZGJOI2RMLMUT7EVJNSXQ53BB4G5QMHEUREKNW23L4V3LQ, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.738045878 (509h13m55.609128913s) [noise=true , address=145.79.227.25:4618 , piece_id=S7VAIMY5CMJIU7PYO66XVBQGZIAYNWWIVQYEO5PCK4774A5CYBCA, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.738454646 (509h13m55.609537691s) [noise=true , address=145.79.227.36:4517 , piece_id=2HDK2GDQ3V7JG3SCR235TJE6AETIWD4TCXOIWH5K23WJGTB7WABA, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.739559450 (509h13m55.610642485s) [noise=true , address=145.79.227.42:4123 , piece_id=SI2NCHXTYUN7MFEIWUDSPIUODD3O7JLCW6IZ5CPE5HVIC2EIX3LA, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.767277460 (509h13m55.638360495s) [noise=true , address=145.79.227.13:4804 , piece_id=WU7S7LN3IPTR2HFTSKLEXDMMQMYC4BDSCQKFIHKTXUH4YK3CBQTQ, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:55.986559662 (509h13m55.857642697s) [noise=true , address=45.91.116.10:4083 , piece_id=VZ7HEFNEDRYHBCLYVDGROGXN7KP4274HCA5KKMKC66LFDPWMQX4A, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:56.009593071 (509h13m55.880676116s) [noise=true , address=45.91.116.12:4226 , piece_id=5RRTTAKVHXJNIW2B3EJNG654W76OKRP5SJT66UPTMYQ3RYBHNLFQ, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:56.014253576 (509h13m55.885336611s) [noise=true , address=77.90.31.10:4520 , piece_id=EVOR62CPL63GGJJQA6GXOA2WAW3PG32YXCRB77MFQNEVSDH4P24A, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:56.194939815 (509h13m56.06602286s) [noise=true , address=77.90.31.14:4661 , piece_id=PUXUG2OO77TB2VWYGABS4K7NMSFGLUG77VOBZKJDGNRGSFIDT7FQ, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:57.019585927 (509h13m56.890668972s) [noise=true , address=45.91.116.6:4974 , piece_id=P4MI4VNKCMSC56YNO52E623IXIUAKP2LNBEG7T3OP5YNKKMX7BCQ, seg_pos={0 0}, upload=446850]: Done uploading piece. err: <nil> Dec 26 20:42:57.019604753 (509h13m56.890687798s) [seg_pos={0 0}, upload=446850]: Segment reached optimal threshold of 13 pieces. Dec 26 20:42:57.019808722 (509h13m56.890891787s) [noise=true , address=145.79.227.3:4150 , piece_id=KIBZZD5MLGRMOEZMFBQODTEDF7CPDTFRN74BQ3R7WVGAOA2ZY55Q, seg_pos={0 0}, upload=446850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 26 20:42:57.019819462 (509h13m56.890902507s) [noise=true , address=145.79.227.3:4150 , piece_id=KIBZZD5MLGRMOEZMFBQODTEDF7CPDTFRN74BQ3R7WVGAOA2ZY55Q, seg_pos={0 0}, upload=446850]: Long tail cancellation due to optimal threshold Dec 26 20:42:57.020170470 (509h13m56.891253515s) [noise=true , address=145.79.227.44:4195 , piece_id=5M56H5P3H4W2PV3VFFPV33GT7TVSKHICJQ2BC5JOADZRKEIEDZDQ, seg_pos={0 0}, upload=446850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 20:42:57.020178455 (509h13m56.89126149s) [noise=true , address=145.79.227.44:4195 , piece_id=5M56H5P3H4W2PV3VFFPV33GT7TVSKHICJQ2BC5JOADZRKEIEDZDQ, seg_pos={0 0}, upload=446850]: Long tail cancellation due to optimal threshold Dec 26 20:42:57.020336396 (509h13m56.891419441s) [noise=true , address=145.79.227.4:4186 , piece_id=TVEWTXJF5P5HH53BHBNVUIU7EZ7LFG4XQYJEUZ46IHGOIWRBGBVA, seg_pos={0 0}, upload=446850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:57302->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:57302->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57302->145.79.227.4:4186: use of closed network connection Dec 26 20:42:57.020345123 (509h13m56.891428168s) [noise=true , address=145.79.227.4:4186 , piece_id=TVEWTXJF5P5HH53BHBNVUIU7EZ7LFG4XQYJEUZ46IHGOIWRBGBVA, seg_pos={0 0}, upload=446850]: Long tail cancellation due to optimal threshold Dec 26 20:42:57.020366534 (509h13m56.891449579s) [seg_pos={0 0}, upload=446850]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:42:57.020442278 (509h13m56.891525323s) [upload=446850]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:42:57.020447237 (509h13m56.891530272s) [upload=446850]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:42:57.306538346 (509h13m57.177621401s) [upload=446850]: Done uploading segments... Dec 26 20:42:57.306563945 (509h13m57.17764699s) [upload=446850]: Upload finished. err: <nil> Duration: 30.002624461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..............X.....................] duration: 30.345046084s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:43:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 30.345063429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X..............X....................] duration: 15.510529434s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:42:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 15.510655675s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..] duration: 6.210911421s trace: https://jaeger.dev.storj.io/trace/61359e2ec8c725f4 FAIL (2025-12-26 20:42:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ) context deadline exceeded ======LOGS====== segments... Dec 26 20:41:18.018143223 (509h12m17.889226278s) [upload=446828]: Got next segment. Inline: false Dec 26 20:41:18.018170776 (509h12m17.889253811s) [upload=446828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:41:18.018175695 (509h12m17.88925874s) [upload=446828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:41:18.268182603 (509h12m18.139265638s) [seg_pos={0 0}, upload=446828]: Begin upload segment... Dec 26 20:41:18.268227127 (509h12m18.139310172s) [seg_pos={0 0}, upload=446828]: Done begin upload segment. Dec 26 20:41:18.268252105 (509h12m18.13933514s) [noise=true , address=145.79.227.50:4374 , piece_id=VPVCM7DQ3KA3CMCHP2PTNXQTRXBFZWLQNBMIII6RGGW76LFGZIBA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.268324282 (509h12m18.139407317s) [noise=true , address=45.91.116.13:4153 , piece_id=QQ6CNEREFND74LQ5HBUNSYNJYIT2CGR4FADQMHOM6UGBBNO2PS4Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.268882405 (509h12m18.13996545s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.269337542 (509h12m18.140420587s) [noise=true , address=145.79.227.46:4603 , piece_id=FTWAMIJRT4P2EEXFSHT45AK4T5VQHRK6Q7GRWXUU2UOBYZG2W4QQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.269733095 (509h12m18.14081613s) [noise=true , address=45.91.116.6:4975 , piece_id=DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.270201988 (509h12m18.141285023s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.270705347 (509h12m18.141788382s) [noise=true , address=145.79.227.17:4971 , piece_id=EL765BLGLOXILLISPRPGK2Z3Q2TTJPS6FDASD6I3V4PARFBSEPKA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.271129855 (509h12m18.1422129s) [noise=true , address=145.79.227.11:4732 , piece_id=QF3YO6ETHVSMC7ZUPFK2D6DH6GSYDHJWRHR5DEKL72WWZFMVJU3Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.272047201 (509h12m18.143130236s) [noise=true , address=145.79.227.36:4517 , piece_id=RW5XTZ5P5MXYDRGM2DMBGAJNJGO2RYV7DGHADJDGTL6GNKDTDLGA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.273169739 (509h12m18.144252784s) [noise=true , address=145.79.227.29:4942 , piece_id=Q7JMTLQ3DM4PZGHDMXH5CJ3LAS3S5HSB276XWCFP6MF6FVDVYYTQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.273302792 (509h12m18.144385837s) [noise=true , address=77.90.31.14:4663 , piece_id=VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.276011821 (509h12m18.147094856s) [noise=true , address=145.79.227.48:4302 , piece_id=HG5LF6OM7ORUEOSP34Y7U43ENNB4M2QPZMC6Q3IKHGTQFJSJRGOQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.276174972 (509h12m18.147258007s) [noise=true , address=45.91.116.7:4012 , piece_id=FTDB5D2ELDJ2KZYSYVVTHWGXG2WM77ELTKISQZLABOT6XIBEG6EQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.279685707 (509h12m18.150768752s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.283723977 (509h12m18.154807022s) [noise=true , address=145.79.227.39:4013 , piece_id=5YYPLG47BFMHX4UZ5L652B67OLST64PSNJUVQEHIAO4MNSWRPC6Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.284762333 (509h12m18.155845378s) [noise=true , address=145.79.227.45:4229 , piece_id=MU5KNWCW3J3R5SOA7KP5UYL4DLUS3GOEPUJHGEMYPCG7EACGZGMA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:19.988111673 (509h12m19.859194738s) [upload=446828]: Next returned nil segment Dec 26 20:41:19.988143443 (509h12m19.859226488s) [upload=446828]: Waiting for error group managing segments... Dec 26 20:41:20.276877755 (509h12m20.14796079s) [noise=true , address=145.79.227.45:4229 , piece_id=MU5KNWCW3J3R5SOA7KP5UYL4DLUS3GOEPUJHGEMYPCG7EACGZGMA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.278237724 (509h12m20.149320759s) [noise=true , address=145.79.227.29:4942 , piece_id=Q7JMTLQ3DM4PZGHDMXH5CJ3LAS3S5HSB276XWCFP6MF6FVDVYYTQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.279899659 (509h12m20.150982694s) [noise=true , address=145.79.227.36:4517 , piece_id=RW5XTZ5P5MXYDRGM2DMBGAJNJGO2RYV7DGHADJDGTL6GNKDTDLGA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.281332086 (509h12m20.152415131s) [noise=true , address=145.79.227.46:4603 , piece_id=FTWAMIJRT4P2EEXFSHT45AK4T5VQHRK6Q7GRWXUU2UOBYZG2W4QQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.290793022 (509h12m20.161876077s) [noise=true , address=145.79.227.48:4302 , piece_id=HG5LF6OM7ORUEOSP34Y7U43ENNB4M2QPZMC6Q3IKHGTQFJSJRGOQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.292610823 (509h12m20.163693868s) [noise=true , address=145.79.227.11:4732 , piece_id=QF3YO6ETHVSMC7ZUPFK2D6DH6GSYDHJWRHR5DEKL72WWZFMVJU3Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.296464041 (509h12m20.167547086s) [noise=true , address=145.79.227.39:4013 , piece_id=5YYPLG47BFMHX4UZ5L652B67OLST64PSNJUVQEHIAO4MNSWRPC6Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.307126443 (509h12m20.178209478s) [noise=true , address=145.79.227.50:4374 , piece_id=VPVCM7DQ3KA3CMCHP2PTNXQTRXBFZWLQNBMIII6RGGW76LFGZIBA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.404347300 (509h12m20.275430335s) [noise=true , address=145.79.227.17:4971 , piece_id=EL765BLGLOXILLISPRPGK2Z3Q2TTJPS6FDASD6I3V4PARFBSEPKA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.453137093 (509h12m20.324220128s) [noise=true , address=77.90.31.14:4663 , piece_id=VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.501363175 (509h12m20.37244621s) [noise=true , address=45.91.116.7:4012 , piece_id=FTDB5D2ELDJ2KZYSYVVTHWGXG2WM77ELTKISQZLABOT6XIBEG6EQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.524892048 (509h12m20.395975093s) [noise=true , address=45.91.116.6:4975 , piece_id=DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.615609551 (509h12m20.486692586s) [noise=true , address=45.91.116.13:4153 , piece_id=QQ6CNEREFND74LQ5HBUNSYNJYIT2CGR4FADQMHOM6UGBBNO2PS4Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.615633336 (509h12m20.486716371s) [seg_pos={0 0}, upload=446828]: Segment reached optimal threshold of 13 pieces. Dec 26 20:41:20.615844709 (509h12m20.486927744s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 20:41:20.615852433 (509h12m20.486935478s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.615990236 (509h12m20.487073281s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection Dec 26 20:41:20.615996357 (509h12m20.487079402s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.616372924 (509h12m20.487455969s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection Dec 26 20:41:20.616393494 (509h12m20.487476539s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.616450602 (509h12m20.487533647s) [seg_pos={0 0}, upload=446828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:41:20.616640264 (509h12m20.487723309s) [upload=446828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:41:20.616649211 (509h12m20.487732246s) [upload=446828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:41:20.920505607 (509h12m20.791588662s) [upload=446828]: Done uploading segments... Dec 26 20:41:20.920531566 (509h12m20.791614611s) [upload=446828]: Upload finished. err: <nil> Duration: 6.210932732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X..............X...................] duration: 22.335169513s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:42:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 22.335195602s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX.] duration: 5.985062502s trace: https://jaeger.dev.storj.io/trace/61359e2ec8c725f4 FAIL (2025-12-26 20:41:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ) context deadline exceeded ======LOGS====== segments... Dec 26 20:41:18.018143223 (509h12m17.889226278s) [upload=446828]: Got next segment. Inline: false Dec 26 20:41:18.018170776 (509h12m17.889253811s) [upload=446828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:41:18.018175695 (509h12m17.88925874s) [upload=446828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:41:18.268182603 (509h12m18.139265638s) [seg_pos={0 0}, upload=446828]: Begin upload segment... Dec 26 20:41:18.268227127 (509h12m18.139310172s) [seg_pos={0 0}, upload=446828]: Done begin upload segment. Dec 26 20:41:18.268252105 (509h12m18.13933514s) [noise=true , address=145.79.227.50:4374 , piece_id=VPVCM7DQ3KA3CMCHP2PTNXQTRXBFZWLQNBMIII6RGGW76LFGZIBA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.268324282 (509h12m18.139407317s) [noise=true , address=45.91.116.13:4153 , piece_id=QQ6CNEREFND74LQ5HBUNSYNJYIT2CGR4FADQMHOM6UGBBNO2PS4Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.268882405 (509h12m18.13996545s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.269337542 (509h12m18.140420587s) [noise=true , address=145.79.227.46:4603 , piece_id=FTWAMIJRT4P2EEXFSHT45AK4T5VQHRK6Q7GRWXUU2UOBYZG2W4QQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.269733095 (509h12m18.14081613s) [noise=true , address=45.91.116.6:4975 , piece_id=DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.270201988 (509h12m18.141285023s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.270705347 (509h12m18.141788382s) [noise=true , address=145.79.227.17:4971 , piece_id=EL765BLGLOXILLISPRPGK2Z3Q2TTJPS6FDASD6I3V4PARFBSEPKA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.271129855 (509h12m18.1422129s) [noise=true , address=145.79.227.11:4732 , piece_id=QF3YO6ETHVSMC7ZUPFK2D6DH6GSYDHJWRHR5DEKL72WWZFMVJU3Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.272047201 (509h12m18.143130236s) [noise=true , address=145.79.227.36:4517 , piece_id=RW5XTZ5P5MXYDRGM2DMBGAJNJGO2RYV7DGHADJDGTL6GNKDTDLGA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.273169739 (509h12m18.144252784s) [noise=true , address=145.79.227.29:4942 , piece_id=Q7JMTLQ3DM4PZGHDMXH5CJ3LAS3S5HSB276XWCFP6MF6FVDVYYTQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.273302792 (509h12m18.144385837s) [noise=true , address=77.90.31.14:4663 , piece_id=VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.276011821 (509h12m18.147094856s) [noise=true , address=145.79.227.48:4302 , piece_id=HG5LF6OM7ORUEOSP34Y7U43ENNB4M2QPZMC6Q3IKHGTQFJSJRGOQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.276174972 (509h12m18.147258007s) [noise=true , address=45.91.116.7:4012 , piece_id=FTDB5D2ELDJ2KZYSYVVTHWGXG2WM77ELTKISQZLABOT6XIBEG6EQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.279685707 (509h12m18.150768752s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.283723977 (509h12m18.154807022s) [noise=true , address=145.79.227.39:4013 , piece_id=5YYPLG47BFMHX4UZ5L652B67OLST64PSNJUVQEHIAO4MNSWRPC6Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.284762333 (509h12m18.155845378s) [noise=true , address=145.79.227.45:4229 , piece_id=MU5KNWCW3J3R5SOA7KP5UYL4DLUS3GOEPUJHGEMYPCG7EACGZGMA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:19.988111673 (509h12m19.859194738s) [upload=446828]: Next returned nil segment Dec 26 20:41:19.988143443 (509h12m19.859226488s) [upload=446828]: Waiting for error group managing segments... Dec 26 20:41:20.276877755 (509h12m20.14796079s) [noise=true , address=145.79.227.45:4229 , piece_id=MU5KNWCW3J3R5SOA7KP5UYL4DLUS3GOEPUJHGEMYPCG7EACGZGMA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.278237724 (509h12m20.149320759s) [noise=true , address=145.79.227.29:4942 , piece_id=Q7JMTLQ3DM4PZGHDMXH5CJ3LAS3S5HSB276XWCFP6MF6FVDVYYTQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.279899659 (509h12m20.150982694s) [noise=true , address=145.79.227.36:4517 , piece_id=RW5XTZ5P5MXYDRGM2DMBGAJNJGO2RYV7DGHADJDGTL6GNKDTDLGA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.281332086 (509h12m20.152415131s) [noise=true , address=145.79.227.46:4603 , piece_id=FTWAMIJRT4P2EEXFSHT45AK4T5VQHRK6Q7GRWXUU2UOBYZG2W4QQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.290793022 (509h12m20.161876077s) [noise=true , address=145.79.227.48:4302 , piece_id=HG5LF6OM7ORUEOSP34Y7U43ENNB4M2QPZMC6Q3IKHGTQFJSJRGOQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.292610823 (509h12m20.163693868s) [noise=true , address=145.79.227.11:4732 , piece_id=QF3YO6ETHVSMC7ZUPFK2D6DH6GSYDHJWRHR5DEKL72WWZFMVJU3Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.296464041 (509h12m20.167547086s) [noise=true , address=145.79.227.39:4013 , piece_id=5YYPLG47BFMHX4UZ5L652B67OLST64PSNJUVQEHIAO4MNSWRPC6Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.307126443 (509h12m20.178209478s) [noise=true , address=145.79.227.50:4374 , piece_id=VPVCM7DQ3KA3CMCHP2PTNXQTRXBFZWLQNBMIII6RGGW76LFGZIBA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.404347300 (509h12m20.275430335s) [noise=true , address=145.79.227.17:4971 , piece_id=EL765BLGLOXILLISPRPGK2Z3Q2TTJPS6FDASD6I3V4PARFBSEPKA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.453137093 (509h12m20.324220128s) [noise=true , address=77.90.31.14:4663 , piece_id=VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.501363175 (509h12m20.37244621s) [noise=true , address=45.91.116.7:4012 , piece_id=FTDB5D2ELDJ2KZYSYVVTHWGXG2WM77ELTKISQZLABOT6XIBEG6EQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.524892048 (509h12m20.395975093s) [noise=true , address=45.91.116.6:4975 , piece_id=DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.615609551 (509h12m20.486692586s) [noise=true , address=45.91.116.13:4153 , piece_id=QQ6CNEREFND74LQ5HBUNSYNJYIT2CGR4FADQMHOM6UGBBNO2PS4Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.615633336 (509h12m20.486716371s) [seg_pos={0 0}, upload=446828]: Segment reached optimal threshold of 13 pieces. Dec 26 20:41:20.615844709 (509h12m20.486927744s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 20:41:20.615852433 (509h12m20.486935478s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.615990236 (509h12m20.487073281s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection Dec 26 20:41:20.615996357 (509h12m20.487079402s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.616372924 (509h12m20.487455969s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection Dec 26 20:41:20.616393494 (509h12m20.487476539s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.616450602 (509h12m20.487533647s) [seg_pos={0 0}, upload=446828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:41:20.616640264 (509h12m20.487723309s) [upload=446828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:41:20.616649211 (509h12m20.487732246s) [upload=446828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:41:20.920505607 (509h12m20.791588662s) [upload=446828]: Done uploading segments... Dec 26 20:41:20.920531566 (509h12m20.791614611s) [upload=446828]: Upload finished. err: <nil> Duration: 5.985081719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.XX] duration: 30.003450872s trace: https://jaeger.dev.storj.io/trace/61359e2ec8c725f4 FAIL (2025-12-26 20:41:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ) context deadline exceeded ======LOGS====== segments... Dec 26 20:41:18.018143223 (509h12m17.889226278s) [upload=446828]: Got next segment. Inline: false Dec 26 20:41:18.018170776 (509h12m17.889253811s) [upload=446828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:41:18.018175695 (509h12m17.88925874s) [upload=446828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:41:18.268182603 (509h12m18.139265638s) [seg_pos={0 0}, upload=446828]: Begin upload segment... Dec 26 20:41:18.268227127 (509h12m18.139310172s) [seg_pos={0 0}, upload=446828]: Done begin upload segment. Dec 26 20:41:18.268252105 (509h12m18.13933514s) [noise=true , address=145.79.227.50:4374 , piece_id=VPVCM7DQ3KA3CMCHP2PTNXQTRXBFZWLQNBMIII6RGGW76LFGZIBA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.268324282 (509h12m18.139407317s) [noise=true , address=45.91.116.13:4153 , piece_id=QQ6CNEREFND74LQ5HBUNSYNJYIT2CGR4FADQMHOM6UGBBNO2PS4Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.268882405 (509h12m18.13996545s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.269337542 (509h12m18.140420587s) [noise=true , address=145.79.227.46:4603 , piece_id=FTWAMIJRT4P2EEXFSHT45AK4T5VQHRK6Q7GRWXUU2UOBYZG2W4QQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.269733095 (509h12m18.14081613s) [noise=true , address=45.91.116.6:4975 , piece_id=DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.270201988 (509h12m18.141285023s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.270705347 (509h12m18.141788382s) [noise=true , address=145.79.227.17:4971 , piece_id=EL765BLGLOXILLISPRPGK2Z3Q2TTJPS6FDASD6I3V4PARFBSEPKA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.271129855 (509h12m18.1422129s) [noise=true , address=145.79.227.11:4732 , piece_id=QF3YO6ETHVSMC7ZUPFK2D6DH6GSYDHJWRHR5DEKL72WWZFMVJU3Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.272047201 (509h12m18.143130236s) [noise=true , address=145.79.227.36:4517 , piece_id=RW5XTZ5P5MXYDRGM2DMBGAJNJGO2RYV7DGHADJDGTL6GNKDTDLGA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.273169739 (509h12m18.144252784s) [noise=true , address=145.79.227.29:4942 , piece_id=Q7JMTLQ3DM4PZGHDMXH5CJ3LAS3S5HSB276XWCFP6MF6FVDVYYTQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.273302792 (509h12m18.144385837s) [noise=true , address=77.90.31.14:4663 , piece_id=VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.276011821 (509h12m18.147094856s) [noise=true , address=145.79.227.48:4302 , piece_id=HG5LF6OM7ORUEOSP34Y7U43ENNB4M2QPZMC6Q3IKHGTQFJSJRGOQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.276174972 (509h12m18.147258007s) [noise=true , address=45.91.116.7:4012 , piece_id=FTDB5D2ELDJ2KZYSYVVTHWGXG2WM77ELTKISQZLABOT6XIBEG6EQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.279685707 (509h12m18.150768752s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.283723977 (509h12m18.154807022s) [noise=true , address=145.79.227.39:4013 , piece_id=5YYPLG47BFMHX4UZ5L652B67OLST64PSNJUVQEHIAO4MNSWRPC6Q, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:18.284762333 (509h12m18.155845378s) [noise=true , address=145.79.227.45:4229 , piece_id=MU5KNWCW3J3R5SOA7KP5UYL4DLUS3GOEPUJHGEMYPCG7EACGZGMA, seg_pos={0 0}, upload=446828]: Uploading piece... Dec 26 20:41:19.988111673 (509h12m19.859194738s) [upload=446828]: Next returned nil segment Dec 26 20:41:19.988143443 (509h12m19.859226488s) [upload=446828]: Waiting for error group managing segments... Dec 26 20:41:20.276877755 (509h12m20.14796079s) [noise=true , address=145.79.227.45:4229 , piece_id=MU5KNWCW3J3R5SOA7KP5UYL4DLUS3GOEPUJHGEMYPCG7EACGZGMA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.278237724 (509h12m20.149320759s) [noise=true , address=145.79.227.29:4942 , piece_id=Q7JMTLQ3DM4PZGHDMXH5CJ3LAS3S5HSB276XWCFP6MF6FVDVYYTQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.279899659 (509h12m20.150982694s) [noise=true , address=145.79.227.36:4517 , piece_id=RW5XTZ5P5MXYDRGM2DMBGAJNJGO2RYV7DGHADJDGTL6GNKDTDLGA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.281332086 (509h12m20.152415131s) [noise=true , address=145.79.227.46:4603 , piece_id=FTWAMIJRT4P2EEXFSHT45AK4T5VQHRK6Q7GRWXUU2UOBYZG2W4QQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.290793022 (509h12m20.161876077s) [noise=true , address=145.79.227.48:4302 , piece_id=HG5LF6OM7ORUEOSP34Y7U43ENNB4M2QPZMC6Q3IKHGTQFJSJRGOQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.292610823 (509h12m20.163693868s) [noise=true , address=145.79.227.11:4732 , piece_id=QF3YO6ETHVSMC7ZUPFK2D6DH6GSYDHJWRHR5DEKL72WWZFMVJU3Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.296464041 (509h12m20.167547086s) [noise=true , address=145.79.227.39:4013 , piece_id=5YYPLG47BFMHX4UZ5L652B67OLST64PSNJUVQEHIAO4MNSWRPC6Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.307126443 (509h12m20.178209478s) [noise=true , address=145.79.227.50:4374 , piece_id=VPVCM7DQ3KA3CMCHP2PTNXQTRXBFZWLQNBMIII6RGGW76LFGZIBA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.404347300 (509h12m20.275430335s) [noise=true , address=145.79.227.17:4971 , piece_id=EL765BLGLOXILLISPRPGK2Z3Q2TTJPS6FDASD6I3V4PARFBSEPKA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.453137093 (509h12m20.324220128s) [noise=true , address=77.90.31.14:4663 , piece_id=VIVTOTN6MYLY7EPE6P6ITUZC7SHBTS7Y7PAVSAPMX35XOIXN2HWQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.501363175 (509h12m20.37244621s) [noise=true , address=45.91.116.7:4012 , piece_id=FTDB5D2ELDJ2KZYSYVVTHWGXG2WM77ELTKISQZLABOT6XIBEG6EQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.524892048 (509h12m20.395975093s) [noise=true , address=45.91.116.6:4975 , piece_id=DZBLBRA4G35PFYRP5D3LOLRLK75H6FLVPBJ3EELKY3R6NGUXXVVA, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.615609551 (509h12m20.486692586s) [noise=true , address=45.91.116.13:4153 , piece_id=QQ6CNEREFND74LQ5HBUNSYNJYIT2CGR4FADQMHOM6UGBBNO2PS4Q, seg_pos={0 0}, upload=446828]: Done uploading piece. err: <nil> Dec 26 20:41:20.615633336 (509h12m20.486716371s) [seg_pos={0 0}, upload=446828]: Segment reached optimal threshold of 13 pieces. Dec 26 20:41:20.615844709 (509h12m20.486927744s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 20:41:20.615852433 (509h12m20.486935478s) [noise=true , address=45.91.116.9:4261 , piece_id=YSATBQFPDJ75SHNALSVOOGJDQJSE4DWA3OL7LNV74FQLE3WE6XIQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.615990236 (509h12m20.487073281s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33910->45.91.116.14:4191: use of closed network connection Dec 26 20:41:20.615996357 (509h12m20.487079402s) [noise=true , address=45.91.116.14:4191 , piece_id=YPFP7JEA5J62PWFUHWEJJMWSBWMMZSPZEUDMEVAENLOFQXIUI2UQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.616372924 (509h12m20.487455969s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33040->145.79.227.2:4113: use of closed network connection Dec 26 20:41:20.616393494 (509h12m20.487476539s) [noise=true , address=145.79.227.2:4113 , piece_id=23R6BBQIPPT6YFQ4SKVRHY3COO3UXFIUTPT5GVDFSE2VNO3744RQ, seg_pos={0 0}, upload=446828]: Long tail cancellation due to optimal threshold Dec 26 20:41:20.616450602 (509h12m20.487533647s) [seg_pos={0 0}, upload=446828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:41:20.616640264 (509h12m20.487723309s) [upload=446828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:41:20.616649211 (509h12m20.487732246s) [upload=446828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:41:20.920505607 (509h12m20.791588662s) [upload=446828]: Done uploading segments... Dec 26 20:41:20.920531566 (509h12m20.791614611s) [upload=446828]: Upload finished. err: <nil> Duration: 30.00348068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X..............X..................] duration: 31.159927842s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:41:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 31.159951747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX..X..X...........XXX.X.X....X..XXXX.X...X.X...XX.XXX...X.X] duration: 30.002294024s trace: https://jaeger.dev.storj.io/trace/27ac1dc1ce614769 FAIL (2025-12-26 20:41:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1522 (509h11m42.833344557s) [upload=446820]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:40:43.211387092 (509h11m43.082470137s) [seg_pos={0 0}, upload=446820]: Begin upload segment... Dec 26 20:40:43.211443689 (509h11m43.082526734s) [seg_pos={0 0}, upload=446820]: Done begin upload segment. Dec 26 20:40:43.211476922 (509h11m43.082559967s) [noise=true , address=145.79.227.48:4302 , piece_id=REW45FGQLXKW6WWY4ZCSBWR2GMFHR7MSIMHZQ2FRDOFEZBDVPE5A, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.211499305 (509h11m43.08258235s) [noise=true , address=77.90.31.10:4520 , piece_id=O4ITDULB7XDQ4NKZVN4YZS5GFBEIQWCC2CBRHFFT64GZZFOJ2URA, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.212051827 (509h11m43.083134862s) [noise=true , address=145.79.227.5:4222 , piece_id=Q4PK7NXTVZYJ6XB64SOREQUXBGOU5GH6FVJOJQH7XPBTDC5JWKJQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.212418826 (509h11m43.083501861s) [noise=true , address=45.91.116.10:4082 , piece_id=K3NBD25BBARUWDZNIZ4XNICBN7QYNNIWN4Y6C2US4LPDBMZOLXRQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.212783811 (509h11m43.083866856s) [noise=true , address=145.79.227.36:4517 , piece_id=BHUS4UP2EVEVAN7WDCFGG7M4VQHFRDUIWPSB4SRA3PQ427LAKZTQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.213151511 (509h11m43.084234556s) [noise=true , address=45.91.116.6:4975 , piece_id=4HIA72JC65B3XXCN3JLLQDN3SWHVQQLXMAST5CWFTVL4KCW7QM5A, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.213555791 (509h11m43.084638836s) [noise=true , address=45.91.116.13:4153 , piece_id=WI7ZN2RW3BLARRAK5HPLMFYG4KSM33TSLTX5YANIHQFWWHPZVUSA, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.214484630 (509h11m43.085567665s) [noise=true , address=145.79.227.41:4086 , piece_id=ZYEKA24NWBT5LIL44KEDW2AF6UDDLHJPJNGKY2A5GKRZLKRQVFIQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.214784019 (509h11m43.085867064s) [noise=true , address=77.90.31.17:4773 , piece_id=Q45DXROJS756ZMPYB4DSGGC2B7VFHM3Y477MEI3DKOJDW2LYTYWQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.215371789 (509h11m43.086454824s) [noise=true , address=145.79.227.45:4230 , piece_id=7NOZJPCAHQXGMBNZVGDPC6FD4YJGUVB6XYHMIAXJZEQSMIH6LJFA, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.215776399 (509h11m43.086859435s) [noise=true , address=145.79.227.11:4730 , piece_id=T3QT2JV6LOQSWS5WVLGBYGX4XFKVPC5H3EXFRUYM4LPSQTF3HTVQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.216616659 (509h11m43.087699704s) [noise=true , address=145.79.227.46:4601 , piece_id=3X5YTJUYGKNHLR7BAU7XKIPZJ6AL6ZU7CCHD6YJ6S54IC45HNMLQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.217114858 (509h11m43.088197893s) [noise=true , address=145.79.227.49:4339 , piece_id=4LXJMUSX75BU35N2HDZGUWQ4A44XMO2PKFHRKUKLMCMPXRE6X7GA, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.218015643 (509h11m43.089098678s) [noise=true , address=145.79.227.6:4259 , piece_id=6QTBYCHQIKYTRRW7KBQ75Q7E4D4L23CAIZJEDZTXPJZLXRQCJDIQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.219325618 (509h11m43.090408653s) [noise=true , address=145.79.227.52:4446 , piece_id=EUGTKXFSSZAIVO2LDNM6NOSI2CXJTW2RYO4A6ZZTFELPBNH2AM2A, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:43.220749509 (509h11m43.091832554s) [noise=true , address=45.91.116.8:4047 , piece_id=733PERYUX63YGKH7LIJE2BJQUAS5XE4IQ7IBO36DBHTIBXOE2MFQ, seg_pos={0 0}, upload=446820]: Uploading piece... Dec 26 20:40:44.930623463 (509h11m44.801706498s) [upload=446820]: Next returned nil segment Dec 26 20:40:44.930638151 (509h11m44.801721186s) [upload=446820]: Waiting for error group managing segments... Dec 26 20:40:45.210619666 (509h11m45.081702711s) [noise=true , address=145.79.227.49:4339 , piece_id=4LXJMUSX75BU35N2HDZGUWQ4A44XMO2PKFHRKUKLMCMPXRE6X7GA, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.217776223 (509h11m45.088859258s) [noise=true , address=145.79.227.48:4302 , piece_id=REW45FGQLXKW6WWY4ZCSBWR2GMFHR7MSIMHZQ2FRDOFEZBDVPE5A, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.224708554 (509h11m45.095791589s) [noise=true , address=145.79.227.45:4230 , piece_id=7NOZJPCAHQXGMBNZVGDPC6FD4YJGUVB6XYHMIAXJZEQSMIH6LJFA, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.228149926 (509h11m45.099232961s) [noise=true , address=145.79.227.36:4517 , piece_id=BHUS4UP2EVEVAN7WDCFGG7M4VQHFRDUIWPSB4SRA3PQ427LAKZTQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.228965359 (509h11m45.100048394s) [noise=true , address=145.79.227.46:4601 , piece_id=3X5YTJUYGKNHLR7BAU7XKIPZJ6AL6ZU7CCHD6YJ6S54IC45HNMLQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.230541060 (509h11m45.101624095s) [noise=true , address=145.79.227.11:4730 , piece_id=T3QT2JV6LOQSWS5WVLGBYGX4XFKVPC5H3EXFRUYM4LPSQTF3HTVQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.256049983 (509h11m45.127133018s) [noise=true , address=145.79.227.52:4446 , piece_id=EUGTKXFSSZAIVO2LDNM6NOSI2CXJTW2RYO4A6ZZTFELPBNH2AM2A, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.319631223 (509h11m45.190714268s) [noise=true , address=145.79.227.6:4259 , piece_id=6QTBYCHQIKYTRRW7KBQ75Q7E4D4L23CAIZJEDZTXPJZLXRQCJDIQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.460268812 (509h11m45.331351858s) [noise=true , address=45.91.116.10:4082 , piece_id=K3NBD25BBARUWDZNIZ4XNICBN7QYNNIWN4Y6C2US4LPDBMZOLXRQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.513139858 (509h11m45.384222903s) [noise=true , address=77.90.31.10:4520 , piece_id=O4ITDULB7XDQ4NKZVN4YZS5GFBEIQWCC2CBRHFFT64GZZFOJ2URA, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.522104709 (509h11m45.393187754s) [noise=true , address=45.91.116.13:4153 , piece_id=WI7ZN2RW3BLARRAK5HPLMFYG4KSM33TSLTX5YANIHQFWWHPZVUSA, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.528016115 (509h11m45.39909916s) [noise=true , address=45.91.116.8:4047 , piece_id=733PERYUX63YGKH7LIJE2BJQUAS5XE4IQ7IBO36DBHTIBXOE2MFQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.568645140 (509h11m45.439728175s) [noise=true , address=45.91.116.6:4975 , piece_id=4HIA72JC65B3XXCN3JLLQDN3SWHVQQLXMAST5CWFTVL4KCW7QM5A, seg_pos={0 0}, upload=446820]: Done uploading piece. err: <nil> Dec 26 20:40:45.568665869 (509h11m45.439748904s) [seg_pos={0 0}, upload=446820]: Segment reached optimal threshold of 13 pieces. Dec 26 20:40:45.568876941 (509h11m45.439959976s) [noise=true , address=145.79.227.41:4086 , piece_id=ZYEKA24NWBT5LIL44KEDW2AF6UDDLHJPJNGKY2A5GKRZLKRQVFIQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:33010->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:33010->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33010->145.79.227.41:4086: use of closed network connection Dec 26 20:40:45.568883745 (509h11m45.43996678s) [noise=true , address=145.79.227.41:4086 , piece_id=ZYEKA24NWBT5LIL44KEDW2AF6UDDLHJPJNGKY2A5GKRZLKRQVFIQ, seg_pos={0 0}, upload=446820]: Long tail cancellation due to optimal threshold Dec 26 20:40:45.568894275 (509h11m45.43997731s) [noise=true , address=145.79.227.5:4222 , piece_id=Q4PK7NXTVZYJ6XB64SOREQUXBGOU5GH6FVJOJQH7XPBTDC5JWKJQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:43704->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:43704->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43704->145.79.227.5:4222: use of closed network connection Dec 26 20:40:45.568902871 (509h11m45.439985906s) [noise=true , address=145.79.227.5:4222 , piece_id=Q4PK7NXTVZYJ6XB64SOREQUXBGOU5GH6FVJOJQH7XPBTDC5JWKJQ, seg_pos={0 0}, upload=446820]: Long tail cancellation due to optimal threshold Dec 26 20:40:45.569000908 (509h11m45.440083943s) [noise=true , address=77.90.31.17:4773 , piece_id=Q45DXROJS756ZMPYB4DSGGC2B7VFHM3Y477MEI3DKOJDW2LYTYWQ, seg_pos={0 0}, upload=446820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:57046->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:57046->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57046->77.90.31.17:4773: use of closed network connection Dec 26 20:40:45.569007911 (509h11m45.440090946s) [noise=true , address=77.90.31.17:4773 , piece_id=Q45DXROJS756ZMPYB4DSGGC2B7VFHM3Y477MEI3DKOJDW2LYTYWQ, seg_pos={0 0}, upload=446820]: Long tail cancellation due to optimal threshold Dec 26 20:40:45.569035594 (509h11m45.440118629s) [seg_pos={0 0}, upload=446820]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:40:45.569117159 (509h11m45.440200194s) [upload=446820]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:40:45.569122008 (509h11m45.440205053s) [upload=446820]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:40:45.858244249 (509h11m45.729327284s) [upload=446820]: Done uploading segments... Dec 26 20:40:45.858273784 (509h11m45.729356819s) [upload=446820]: Upload finished. err: <nil> Duration:* 30.002315424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X..............X.................] duration: 8.888216285s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:40:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 8.888230843s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)