Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration: 26.04496572s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:29:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 26.045075841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 15.531152878s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:28:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 15.531181552s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.742395698s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:28:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 5.742422318s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.924743952s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:27:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 5.924774329s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 15.128305311s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:27:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 15.128332241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.893194557s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:27:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 5.893217741s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.047205542s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:27:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 6.047238343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 15.446260549s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:26:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 15.446288923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.070112162s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:26:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 6.070140817s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 10.589281322s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:25:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 10.58930588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 19.590050008s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:25:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 19.590079205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.845963901s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:25:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 5.845997695s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 20.325197914s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:25:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 20.325230827s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 33.868538755s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:24:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 33.868554054s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.817197073s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:24:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 5.817228875s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 39.045885937s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:24:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 39.045937786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.725810074s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:23:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 5.725840542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 16.307658696s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:23:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 16.307676309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 40.838046862s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:22:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 40.83806165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.740963945s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:22:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 5.740990635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 1m0.003446348s trace: https://jaeger.dev.storj.io/trace/2dde5fb84d6f9aef FAIL (2025-12-28 22:22:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: 7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA) context deadline exceeded; piecestore: (Node ID: 1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, Piece ID: TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA) context deadline exceeded ======LOGS====== Done begin upload segment. Dec 28 22:21:13.466126077 (558h52m13.337209112s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466130635 (558h52m13.33721367s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466584950 (558h52m13.337667985s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466681324 (558h52m13.337764369s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.466954685 (558h52m13.33803772s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467283190 (558h52m13.338366235s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467651912 (558h52m13.338734957s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.467926144 (558h52m13.33900918s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468224863 (558h52m13.339307908s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468533250 (558h52m13.339616295s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.468852889 (558h52m13.339935924s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.474952334 (558h52m13.346035369s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.476142490 (558h52m13.347225525s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.477282110 (558h52m13.348365145s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.478719176 (558h52m13.349802211s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.479894605 (558h52m13.35097765s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.481013675 (558h52m13.35209671s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:13.482208331 (558h52m13.353291366s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Uploading piece... Dec 28 22:21:14.924931350 (558h52m14.796014395s) [upload=489964]: Next returned nil segment Dec 28 22:21:14.924963901 (558h52m14.796046966s) [upload=489964]: Waiting for error group managing segments... Dec 28 22:21:15.127660355 (558h52m14.99874339s) [noise=true , address=193.149.164.143:4700 , piece_id=5PWWM25RRXBVI22OK6NUY5456WYUS5NXCM37CIUQH42RWFAXZNVQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Dec 28 22:21:15.345729562 (558h52m15.216812607s) [noise=true , address=145.79.227.46:4633 , piece_id=YHGVFZ4FSPUTHZ7EEVG44H7BOUR57HCZ5KBADKYHWAP4ACU6JXYA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.353583838 (558h52m15.224666873s) [noise=true , address=145.79.227.42:4133 , piece_id=AHEPSOKVANAARMT2ARGBC4GFVNZSO3GCMRUNTEKVM4HV7OVGT5VA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.491284908 (558h52m15.362367943s) [noise=true , address=77.90.31.18:4821 , piece_id=7ZVCTZ6AWFFC24OMGWQYICT5HRAECGYUMZ37Z2L2L2C3YFAYM5OQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.516145146 (558h52m15.387228191s) [noise=true , address=77.90.31.15:4705 , piece_id=2GM5IUEFVF2MTASM7CUDTAQPIDR3TMNK223V53JWIKXUS6ZS77OA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.539545846 (558h52m15.410628901s) [noise=true , address=77.90.31.21:4895 , piece_id=ZPVK5EP7MPKEKT5IOQUVIKDK2OXNWSDGATTMFKOH3E45F2J6YIMA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.549881166 (558h52m15.420964201s) [noise=true , address=45.91.116.12:4256 , piece_id=ZY4HESLIENUYZ7OV3DLQNR5CU6XXIAUMV2SROOOXSIK6EBUX2LJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:15.595277810 (558h52m15.466360855s) [noise=true , address=45.91.116.9:4286 , piece_id=CUVH4J7PBZMHQSAWF7Z5UVYKZA5DYVVYELAGK2BSEOHS3MHSMAPA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.246156712 (558h52m16.117239757s) [noise=true , address=145.79.227.44:4196 , piece_id=EOOZHIAXLANLCV22PBQBP6MZF5KOBJ7IU6ARW6CG5QCRFHZAEJJA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.275617411 (558h52m16.146700456s) [noise=true , address=145.79.227.43:4173 , piece_id=CL5EFMECIML2SM7TSAA2YZLQHJSCCMJ7QP7HL2ZO6KMUNLZEWIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.309349274 (558h52m16.180432319s) [noise=true , address=77.90.31.12:4624 , piece_id=UZJSNXL3KFO4HIIJKQSWRCIAO6H5WEHN5UHUIOJ4ZBTRECLI3T3A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.342178146 (558h52m16.213261191s) [noise=true , address=77.90.31.10:4522 , piece_id=2T36DOELLDK6D7QN6VVJHHA7PHUD2LNSRFB7HGOYKSDX7CXLJERQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.377649129 (558h52m16.248732164s) [noise=true , address=145.79.227.49:4348 , piece_id=5KTPR7LD6SIQELOV4X5BRUGYOQSCKEKP5QW64AVJR24FHPIRMB4A, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.430356415 (558h52m16.30143945s) [noise=true , address=145.79.227.27:4701 , piece_id=SI4W5S7ENQVWWOY4SZ67QYSCRIWRT4VGZKKVMFSCVG2Y7K4J5GEA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559330918 (558h52m16.430413953s) [noise=true , address=193.149.164.143:4704 , piece_id=TXR5XZUGWF6GL2GGSBXLKYEB7M6JFENXNTL7PUIDMT46DAQ66WAA, seg_pos={0 0}, upload=489964]: Done uploading piece. err: <nil> Dec 28 22:21:16.559353802 (558h52m16.430436847s) [seg_pos={0 0}, upload=489964]: Segment reached optimal threshold of 14 pieces. Dec 28 22:21:16.559599279 (558h52m16.430682314s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559607655 (558h52m16.4306907s) [noise=true , address=64.49.13.170:4717 , piece_id=EXMLW5RNAAME45WKZMCLT2UJ3XSWZSPA4J545Y2JTKRWJMQQSIAQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559789661 (558h52m16.430872706s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dgidHuoyZFQZQyBAfQrBAn3FXjgePo8GjuzPyBijKp5xb4wk1): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559810030 (558h52m16.430893065s) [noise=true , address=64.49.13.137:4495 , piece_id=PVNTHBZAQEPVP4MIFVSKLS3D7PCQVIXXY5FTGEWQJKWFJ7TD46SQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559920550 (558h52m16.431003596s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF Dec 28 22:21:16.559930830 (558h52m16.431013875s) [noise=true , address=77.90.31.11:4581 , piece_id=DVXCCJEL55OEO5MHCIKC6FHCJAUCGEZXKRMFXJO4P74PIHCW5QTQ, seg_pos={0 0}, upload=489964]: Long tail cancellation due to optimal threshold Dec 28 22:21:16.559951920 (558h52m16.431034965s) [seg_pos={0 0}, upload=489964]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:21:16.560031201 (558h52m16.431114236s) [upload=489964]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:21:16.560037152 (558h52m16.431120187s) [upload=489964]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:21:16.847427049 (558h52m16.718510084s) [upload=489964]: Done uploading segments... Dec 28 22:21:16.847446086 (558h52m16.718529131s) [upload=489964]: Upload finished. err: <nil> Duration: 1m0.003494018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X................................................X..........] duration: 5.678073064s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:22:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 5.67810873s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X................................................X.........] duration: 6.221501427s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:21:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 6.221526825s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 19.505187651s trace: https://jaeger.dev.storj.io/trace/645110d5e877847 FAIL (2025-12-28 22:20:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.29:4954 , piece_id=2URNLYC6ISJFF63ELKWDDST2KGZFWBXREWI22IVOHKX3QZ2RPULA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.095269840 (558h50m11.966352875s) [noise=true , address=77.90.31.17:4790 , piece_id=22SAO632DHUFFZYSPLV7SUY3KDI2TTLO4GPQJXJE7KCUZILYDBLA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.095574781 (558h50m11.966657816s) [noise=true , address=77.90.31.10:4548 , piece_id=S5K2TBRAWWCEA46VLK6IERLB3MD7GHUAXPRB3RNGHOOXPDWSXBMA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.095929617 (558h50m11.967012652s) [noise=true , address=145.79.227.40:4066 , piece_id=C2I5X2A26KS5SUQNKLPGYEW6XTMTUW6W655EBVI2QU67KIKVNLFA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.096414821 (558h50m11.967497866s) [noise=true , address=77.90.31.12:4597 , piece_id=TDRW5M6YWNU6CIFN6ATYKJQKMK3OLB5IL2BWDK5OGTOBF4VNEMYQ, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.096865699 (558h50m11.967948744s) [noise=true , address=145.79.227.33:4888 , piece_id=V5M7H2HC743PC542S5JGWZOJ4CVAC6A6EQLJQ56SPZ533HCIXDUA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.097359529 (558h50m11.968442564s) [noise=true , address=64.49.13.169:4706 , piece_id=NNAVLHFIWM4XVYRDY6K4WKITHD4YCUNO3JOOFGAOLC7WADBWC5MA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.097866796 (558h50m11.968949831s) [noise=true , address=145.79.227.44:4219 , piece_id=GXZLJK7XXWPVGASDEK66KQRTZO5NBWQZGKVTMLWH3W33VHQKIWKQ, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.097907703 (558h50m11.968990738s) [noise=true , address=64.49.13.154:4926 , piece_id=3HM35YDMF3ZRX7XEN44HO6AOXGZMZQI5MNLVZQWZ2TORSO6GNM5Q, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.098235719 (558h50m11.969318754s) [noise=true , address=145.79.227.49:4361 , piece_id=L7E5X22MVGEK2XO4GRJMPFEV6CIU5JC2QLFSKJRL447I26LUSAZA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.098388158 (558h50m11.969471194s) [noise=true , address=5.231.73.2:4032 , piece_id=AUCBPMMJT2GP56SPQJNHFXQCTRP3EMN7HACPI335ZLKJVFWFCNZQ, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.098617666 (558h50m11.969700711s) [noise=true , address=77.90.31.14:4696 , piece_id=OFLWAZOW6JGDHZ32K5EBF7VHQVKD2KCQWXDH6OWQ6GKLV3BMUIZA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.099057142 (558h50m11.970140187s) [noise=true , address=45.91.116.14:4207 , piece_id=Z6ALMPJ477DUROBM74KTCIJCFIUZCEPU5LGMFM5RVASVJGRGH3PQ, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.099554278 (558h50m11.970637324s) [noise=true , address=45.91.116.9:4281 , piece_id=EKXOBLP7AKWZJDP73CFWOUPYRGVHDH6TKVWIGLKAEA7I722VE3AQ, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.099921628 (558h50m11.971004663s) [noise=true , address=145.79.227.30:4789 , piece_id=CGERHAAWKFRJTPKABIQ7P7PTNG5ZIONORTOQH4ILQXIPGVUTX24A, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.100260875 (558h50m11.97134391s) [noise=true , address=145.79.227.35:4506 , piece_id=AT4RF2FL6XQXLTK6DNGEWF3HLP6Y2EBEY6Q5TITSS6OMAUESLXPA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:13.572791603 (558h50m13.443874638s) [upload=489935]: Next returned nil segment Dec 28 22:19:13.572804668 (558h50m13.443887703s) [upload=489935]: Waiting for error group managing segments... Dec 28 22:19:13.988464987 (558h50m13.859548032s) [noise=true , address=145.79.227.12:4788 , piece_id=YFOE65J4GIPFXJPRMMGL635HR4RQ3UAVMROOGF2CVU2ZWAMVGGZA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.076473704 (558h50m13.947556739s) [noise=true , address=64.49.13.154:4926 , piece_id=3HM35YDMF3ZRX7XEN44HO6AOXGZMZQI5MNLVZQWZ2TORSO6GNM5Q, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.144433803 (558h50m14.015516848s) [noise=true , address=77.90.31.14:4696 , piece_id=OFLWAZOW6JGDHZ32K5EBF7VHQVKD2KCQWXDH6OWQ6GKLV3BMUIZA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.158060561 (558h50m14.029143597s) [noise=true , address=77.90.31.12:4597 , piece_id=TDRW5M6YWNU6CIFN6ATYKJQKMK3OLB5IL2BWDK5OGTOBF4VNEMYQ, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.224892001 (558h50m14.095975036s) [noise=true , address=45.91.116.9:4281 , piece_id=EKXOBLP7AKWZJDP73CFWOUPYRGVHDH6TKVWIGLKAEA7I722VE3AQ, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.226065716 (558h50m14.097148752s) [noise=true , address=45.91.116.14:4207 , piece_id=Z6ALMPJ477DUROBM74KTCIJCFIUZCEPU5LGMFM5RVASVJGRGH3PQ, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.226799634 (558h50m14.097882669s) [noise=true , address=77.90.31.17:4790 , piece_id=22SAO632DHUFFZYSPLV7SUY3KDI2TTLO4GPQJXJE7KCUZILYDBLA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.774911918 (558h50m14.645994953s) [noise=true , address=5.231.73.2:4032 , piece_id=AUCBPMMJT2GP56SPQJNHFXQCTRP3EMN7HACPI335ZLKJVFWFCNZQ, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.853403382 (558h50m14.724486417s) [noise=true , address=145.79.227.44:4219 , piece_id=GXZLJK7XXWPVGASDEK66KQRTZO5NBWQZGKVTMLWH3W33VHQKIWKQ, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.900492238 (558h50m14.771575283s) [noise=true , address=145.79.227.30:4772 , piece_id=H5HQDB7YGAIB6TAJ4FLCALGVNDREED524SC77ACXIBQ7W56AEJNQ, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.901016857 (558h50m14.772099892s) [noise=true , address=145.79.227.33:4888 , piece_id=V5M7H2HC743PC542S5JGWZOJ4CVAC6A6EQLJQ56SPZ533HCIXDUA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.921356077 (558h50m14.792439122s) [noise=true , address=145.79.227.35:4506 , piece_id=AT4RF2FL6XQXLTK6DNGEWF3HLP6Y2EBEY6Q5TITSS6OMAUESLXPA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.936309992 (558h50m14.807393027s) [noise=true , address=145.79.227.30:4789 , piece_id=CGERHAAWKFRJTPKABIQ7P7PTNG5ZIONORTOQH4ILQXIPGVUTX24A, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.991771461 (558h50m14.862854496s) [noise=true , address=77.90.31.10:4548 , piece_id=S5K2TBRAWWCEA46VLK6IERLB3MD7GHUAXPRB3RNGHOOXPDWSXBMA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.991802260 (558h50m14.862885295s) [seg_pos={0 0}, upload=489935]: Segment reached optimal threshold of 14 pieces. Dec 28 22:19:14.992043830 (558h50m14.863126865s) [noise=true , address=145.79.227.49:4361 , piece_id=L7E5X22MVGEK2XO4GRJMPFEV6CIU5JC2QLFSKJRL447I26LUSAZA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 28 22:19:14.992053248 (558h50m14.863136283s) [noise=true , address=145.79.227.49:4361 , piece_id=L7E5X22MVGEK2XO4GRJMPFEV6CIU5JC2QLFSKJRL447I26LUSAZA, seg_pos={0 0}, upload=489935]: Long tail cancellation due to optimal threshold Dec 28 22:19:14.992267756 (558h50m14.863350791s) [noise=true , address=145.79.227.40:4066 , piece_id=C2I5X2A26KS5SUQNKLPGYEW6XTMTUW6W655EBVI2QU67KIKVNLFA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 22:19:14.992283136 (558h50m14.863366171s) [noise=true , address=145.79.227.40:4066 , piece_id=C2I5X2A26KS5SUQNKLPGYEW6XTMTUW6W655EBVI2QU67KIKVNLFA, seg_pos={0 0}, upload=489935]: Long tail cancellation due to optimal threshold Dec 28 22:19:14.992362837 (558h50m14.863445892s) [noise=true , address=64.49.13.169:4706 , piece_id=NNAVLHFIWM4XVYRDY6K4WKITHD4YCUNO3JOOFGAOLC7WADBWC5MA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:37598->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:37598->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37598->64.49.13.169:4706: use of closed network connection Dec 28 22:19:14.992378036 (558h50m14.863461091s) [noise=true , address=64.49.13.169:4706 , piece_id=NNAVLHFIWM4XVYRDY6K4WKITHD4YCUNO3JOOFGAOLC7WADBWC5MA, seg_pos={0 0}, upload=489935]: Long tail cancellation due to optimal threshold Dec 28 22:19:14.992550604 (558h50m14.863633639s) [noise=true , address=145.79.227.29:4954 , piece_id=2URNLYC6ISJFF63ELKWDDST2KGZFWBXREWI22IVOHKX3QZ2RPULA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14F5pJPTejP7BMpo1tcZo6uG9XdxvT7gLAUb1cgeBzKChafSXz): protocol: expected piece hash; context canceled; EOF Dec 28 22:19:14.992559562 (558h50m14.863642597s) [noise=true , address=145.79.227.29:4954 , piece_id=2URNLYC6ISJFF63ELKWDDST2KGZFWBXREWI22IVOHKX3QZ2RPULA, seg_pos={0 0}, upload=489935]: Long tail cancellation due to optimal threshold Dec 28 22:19:14.992580271 (558h50m14.863663316s) [seg_pos={0 0}, upload=489935]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:19:14.992671996 (558h50m14.863755031s) [upload=489935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:19:14.992680372 (558h50m14.863763417s) [upload=489935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:19:15.284646312 (558h50m15.155729347s) [upload=489935]: Done uploading segments... Dec 28 22:19:15.284666982 (558h50m15.155750027s) [upload=489935]: Upload finished. err: <nil> Duration: 19.505217018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X................................................X........] duration: 5.739333397s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:20:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 5.73936064s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.004766173s trace: https://jaeger.dev.storj.io/trace/645110d5e877847 FAIL (2025-12-28 22:20:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.29:4954 , piece_id=2URNLYC6ISJFF63ELKWDDST2KGZFWBXREWI22IVOHKX3QZ2RPULA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.095269840 (558h50m11.966352875s) [noise=true , address=77.90.31.17:4790 , piece_id=22SAO632DHUFFZYSPLV7SUY3KDI2TTLO4GPQJXJE7KCUZILYDBLA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.095574781 (558h50m11.966657816s) [noise=true , address=77.90.31.10:4548 , piece_id=S5K2TBRAWWCEA46VLK6IERLB3MD7GHUAXPRB3RNGHOOXPDWSXBMA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.095929617 (558h50m11.967012652s) [noise=true , address=145.79.227.40:4066 , piece_id=C2I5X2A26KS5SUQNKLPGYEW6XTMTUW6W655EBVI2QU67KIKVNLFA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.096414821 (558h50m11.967497866s) [noise=true , address=77.90.31.12:4597 , piece_id=TDRW5M6YWNU6CIFN6ATYKJQKMK3OLB5IL2BWDK5OGTOBF4VNEMYQ, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.096865699 (558h50m11.967948744s) [noise=true , address=145.79.227.33:4888 , piece_id=V5M7H2HC743PC542S5JGWZOJ4CVAC6A6EQLJQ56SPZ533HCIXDUA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.097359529 (558h50m11.968442564s) [noise=true , address=64.49.13.169:4706 , piece_id=NNAVLHFIWM4XVYRDY6K4WKITHD4YCUNO3JOOFGAOLC7WADBWC5MA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.097866796 (558h50m11.968949831s) [noise=true , address=145.79.227.44:4219 , piece_id=GXZLJK7XXWPVGASDEK66KQRTZO5NBWQZGKVTMLWH3W33VHQKIWKQ, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.097907703 (558h50m11.968990738s) [noise=true , address=64.49.13.154:4926 , piece_id=3HM35YDMF3ZRX7XEN44HO6AOXGZMZQI5MNLVZQWZ2TORSO6GNM5Q, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.098235719 (558h50m11.969318754s) [noise=true , address=145.79.227.49:4361 , piece_id=L7E5X22MVGEK2XO4GRJMPFEV6CIU5JC2QLFSKJRL447I26LUSAZA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.098388158 (558h50m11.969471194s) [noise=true , address=5.231.73.2:4032 , piece_id=AUCBPMMJT2GP56SPQJNHFXQCTRP3EMN7HACPI335ZLKJVFWFCNZQ, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.098617666 (558h50m11.969700711s) [noise=true , address=77.90.31.14:4696 , piece_id=OFLWAZOW6JGDHZ32K5EBF7VHQVKD2KCQWXDH6OWQ6GKLV3BMUIZA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.099057142 (558h50m11.970140187s) [noise=true , address=45.91.116.14:4207 , piece_id=Z6ALMPJ477DUROBM74KTCIJCFIUZCEPU5LGMFM5RVASVJGRGH3PQ, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.099554278 (558h50m11.970637324s) [noise=true , address=45.91.116.9:4281 , piece_id=EKXOBLP7AKWZJDP73CFWOUPYRGVHDH6TKVWIGLKAEA7I722VE3AQ, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.099921628 (558h50m11.971004663s) [noise=true , address=145.79.227.30:4789 , piece_id=CGERHAAWKFRJTPKABIQ7P7PTNG5ZIONORTOQH4ILQXIPGVUTX24A, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:12.100260875 (558h50m11.97134391s) [noise=true , address=145.79.227.35:4506 , piece_id=AT4RF2FL6XQXLTK6DNGEWF3HLP6Y2EBEY6Q5TITSS6OMAUESLXPA, seg_pos={0 0}, upload=489935]: Uploading piece... Dec 28 22:19:13.572791603 (558h50m13.443874638s) [upload=489935]: Next returned nil segment Dec 28 22:19:13.572804668 (558h50m13.443887703s) [upload=489935]: Waiting for error group managing segments... Dec 28 22:19:13.988464987 (558h50m13.859548032s) [noise=true , address=145.79.227.12:4788 , piece_id=YFOE65J4GIPFXJPRMMGL635HR4RQ3UAVMROOGF2CVU2ZWAMVGGZA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.076473704 (558h50m13.947556739s) [noise=true , address=64.49.13.154:4926 , piece_id=3HM35YDMF3ZRX7XEN44HO6AOXGZMZQI5MNLVZQWZ2TORSO6GNM5Q, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.144433803 (558h50m14.015516848s) [noise=true , address=77.90.31.14:4696 , piece_id=OFLWAZOW6JGDHZ32K5EBF7VHQVKD2KCQWXDH6OWQ6GKLV3BMUIZA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.158060561 (558h50m14.029143597s) [noise=true , address=77.90.31.12:4597 , piece_id=TDRW5M6YWNU6CIFN6ATYKJQKMK3OLB5IL2BWDK5OGTOBF4VNEMYQ, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.224892001 (558h50m14.095975036s) [noise=true , address=45.91.116.9:4281 , piece_id=EKXOBLP7AKWZJDP73CFWOUPYRGVHDH6TKVWIGLKAEA7I722VE3AQ, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.226065716 (558h50m14.097148752s) [noise=true , address=45.91.116.14:4207 , piece_id=Z6ALMPJ477DUROBM74KTCIJCFIUZCEPU5LGMFM5RVASVJGRGH3PQ, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.226799634 (558h50m14.097882669s) [noise=true , address=77.90.31.17:4790 , piece_id=22SAO632DHUFFZYSPLV7SUY3KDI2TTLO4GPQJXJE7KCUZILYDBLA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.774911918 (558h50m14.645994953s) [noise=true , address=5.231.73.2:4032 , piece_id=AUCBPMMJT2GP56SPQJNHFXQCTRP3EMN7HACPI335ZLKJVFWFCNZQ, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.853403382 (558h50m14.724486417s) [noise=true , address=145.79.227.44:4219 , piece_id=GXZLJK7XXWPVGASDEK66KQRTZO5NBWQZGKVTMLWH3W33VHQKIWKQ, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.900492238 (558h50m14.771575283s) [noise=true , address=145.79.227.30:4772 , piece_id=H5HQDB7YGAIB6TAJ4FLCALGVNDREED524SC77ACXIBQ7W56AEJNQ, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.901016857 (558h50m14.772099892s) [noise=true , address=145.79.227.33:4888 , piece_id=V5M7H2HC743PC542S5JGWZOJ4CVAC6A6EQLJQ56SPZ533HCIXDUA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.921356077 (558h50m14.792439122s) [noise=true , address=145.79.227.35:4506 , piece_id=AT4RF2FL6XQXLTK6DNGEWF3HLP6Y2EBEY6Q5TITSS6OMAUESLXPA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.936309992 (558h50m14.807393027s) [noise=true , address=145.79.227.30:4789 , piece_id=CGERHAAWKFRJTPKABIQ7P7PTNG5ZIONORTOQH4ILQXIPGVUTX24A, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.991771461 (558h50m14.862854496s) [noise=true , address=77.90.31.10:4548 , piece_id=S5K2TBRAWWCEA46VLK6IERLB3MD7GHUAXPRB3RNGHOOXPDWSXBMA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: <nil> Dec 28 22:19:14.991802260 (558h50m14.862885295s) [seg_pos={0 0}, upload=489935]: Segment reached optimal threshold of 14 pieces. Dec 28 22:19:14.992043830 (558h50m14.863126865s) [noise=true , address=145.79.227.49:4361 , piece_id=L7E5X22MVGEK2XO4GRJMPFEV6CIU5JC2QLFSKJRL447I26LUSAZA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 28 22:19:14.992053248 (558h50m14.863136283s) [noise=true , address=145.79.227.49:4361 , piece_id=L7E5X22MVGEK2XO4GRJMPFEV6CIU5JC2QLFSKJRL447I26LUSAZA, seg_pos={0 0}, upload=489935]: Long tail cancellation due to optimal threshold Dec 28 22:19:14.992267756 (558h50m14.863350791s) [noise=true , address=145.79.227.40:4066 , piece_id=C2I5X2A26KS5SUQNKLPGYEW6XTMTUW6W655EBVI2QU67KIKVNLFA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 22:19:14.992283136 (558h50m14.863366171s) [noise=true , address=145.79.227.40:4066 , piece_id=C2I5X2A26KS5SUQNKLPGYEW6XTMTUW6W655EBVI2QU67KIKVNLFA, seg_pos={0 0}, upload=489935]: Long tail cancellation due to optimal threshold Dec 28 22:19:14.992362837 (558h50m14.863445892s) [noise=true , address=64.49.13.169:4706 , piece_id=NNAVLHFIWM4XVYRDY6K4WKITHD4YCUNO3JOOFGAOLC7WADBWC5MA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:37598->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:37598->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37598->64.49.13.169:4706: use of closed network connection Dec 28 22:19:14.992378036 (558h50m14.863461091s) [noise=true , address=64.49.13.169:4706 , piece_id=NNAVLHFIWM4XVYRDY6K4WKITHD4YCUNO3JOOFGAOLC7WADBWC5MA, seg_pos={0 0}, upload=489935]: Long tail cancellation due to optimal threshold Dec 28 22:19:14.992550604 (558h50m14.863633639s) [noise=true , address=145.79.227.29:4954 , piece_id=2URNLYC6ISJFF63ELKWDDST2KGZFWBXREWI22IVOHKX3QZ2RPULA, seg_pos={0 0}, upload=489935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14F5pJPTejP7BMpo1tcZo6uG9XdxvT7gLAUb1cgeBzKChafSXz): protocol: expected piece hash; context canceled; EOF Dec 28 22:19:14.992559562 (558h50m14.863642597s) [noise=true , address=145.79.227.29:4954 , piece_id=2URNLYC6ISJFF63ELKWDDST2KGZFWBXREWI22IVOHKX3QZ2RPULA, seg_pos={0 0}, upload=489935]: Long tail cancellation due to optimal threshold Dec 28 22:19:14.992580271 (558h50m14.863663316s) [seg_pos={0 0}, upload=489935]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 22:19:14.992671996 (558h50m14.863755031s) [upload=489935]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:19:14.992680372 (558h50m14.863763417s) [upload=489935]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:19:15.284646312 (558h50m15.155729347s) [upload=489935]: Done uploading segments... Dec 28 22:19:15.284666982 (558h50m15.155750027s) [upload=489935]: Upload finished. err: <nil> Duration: 1m0.004779287s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X................................................X.......] duration: 6.04122414s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:19:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 6.041250932s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X................................................X......] duration: 5.931328583s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:18:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 5.931355354s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X................................................X.....] duration: 14.293843277s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:18:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 14.293875197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X................................................X....] duration: 5.893653445s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:17:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 5.893707978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X................................................X...] duration: 5.893593535s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:16:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 5.89370652s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X................................................X..] duration: 5.903866693s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:15:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 5.903895578s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X................................................X.] duration: 6.093909522s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:15:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 6.093936163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X................................................X] duration: 30.002715678s trace: https://jaeger.dev.storj.io/trace/688d13a1f51494bf FAIL (2025-12-28 22:14:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA) context deadline exceeded; piecestore: (Node ID: 12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, Piece ID: SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA) context deadline exceeded ======LOGS====== Dec 28 22:14:22.245952274 (558h45m22.117035309s) [upload=489861]: Starting upload Dec 28 22:14:22.246034361 (558h45m22.117117406s) [upload=489861]: Done starting upload Dec 28 22:14:22.246136726 (558h45m22.117219761s) [upload=489861]: Uploading segments... Dec 28 22:14:22.246394617 (558h45m22.117477652s) [upload=489861]: Got next segment. Inline: false Dec 28 22:14:22.246450664 (558h45m22.117533699s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 22:14:22.246457277 (558h45m22.117540322s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 22:14:22.486108610 (558h45m22.357191645s) [seg_pos={0 0}, upload=489861]: Begin upload segment... Dec 28 22:14:22.486154356 (558h45m22.357237402s) [seg_pos={0 0}, upload=489861]: Done begin upload segment. Dec 28 22:14:22.486172902 (558h45m22.357255937s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486193250 (558h45m22.357276285s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486272201 (558h45m22.357355236s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.486580939 (558h45m22.357663975s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487306971 (558h45m22.358390006s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487605029 (558h45m22.358688064s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.487917785 (558h45m22.35900082s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488226893 (558h45m22.359309928s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488583913 (558h45m22.359666948s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.488956503 (558h45m22.360039538s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.490133394 (558h45m22.361216429s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.491446444 (558h45m22.36252949s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.493023577 (558h45m22.364106622s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.494825869 (558h45m22.365908914s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.497352541 (558h45m22.368435586s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:22.499148450 (558h45m22.370231485s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Uploading piece... Dec 28 22:14:24.197296047 (558h45m24.068379082s) [upload=489861]: Next returned nil segment Dec 28 22:14:24.197311977 (558h45m24.068395012s) [upload=489861]: Waiting for error group managing segments... Dec 28 22:14:24.496129664 (558h45m24.367212699s) [noise=true , address=145.79.227.38:4978 , piece_id=F564VEVWNEI6RNU7LNBHCS5FTGPO6MSROHRLQ5SFDSFPOZGGXESA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498252456 (558h45m24.369335491s) [noise=true , address=145.79.227.4:4185 , piece_id=RQ4ZJFLPBFEAYMD4FRW7WOK7KAASRVDYJTKVSUBS7XJZ62PDCNEA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.498435496 (558h45m24.369518531s) [noise=true , address=145.79.227.19:4077 , piece_id=F6THWPYK4EVMD6FUXW376SZJJHGWQPTS7IRFHYFD6FGQXMN7ESVA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.507174266 (558h45m24.378257311s) [noise=true , address=145.79.227.3:4149 , piece_id=SFVGEZ6U3SGJV3GX4ZSS3TB4IVZOMYASLTO4U4PGYWNX63AFZ7IQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.509763335 (558h45m24.38084637s) [noise=true , address=145.79.227.12:4767 , piece_id=AMBWCIGUW7SWA67EKYTAMYLFEXZILVZNKRWJS4EXEX7OVB23KIOQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.512839362 (558h45m24.383922397s) [noise=true , address=145.79.227.50:4375 , piece_id=5G4MRQB62FLMGVYWIQAGDIU33OY352YXQPXT52EBB7PECI2UKB7A, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.514310995 (558h45m24.38539403s) [noise=true , address=145.79.227.51:4410 , piece_id=D7VNIQ6RUOM6IZQT3S6KNJTY2675KH6RPDON7PTK4PJCQESEFJPA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.518277789 (558h45m24.389360844s) [noise=true , address=145.79.227.7:4293 , piece_id=YCD4TMU4CPSN3DUHZ6QN5ZID26UKZ3CW6AC7VYVJRREI3O33JQCA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.528086897 (558h45m24.399169942s) [noise=true , address=145.79.227.47:4267 , piece_id=WHUELKWDHABZIA4R7PFF76KOCJV5OSHVEH3IBPL6DBYQUIFLWCLA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.530662642 (558h45m24.401745677s) [noise=true , address=145.79.227.39:4014 , piece_id=7XJOVAUQQGTMDKZ2A6JXP4ISLB5YCFH6HS2W45OSOZNV3YS2B5KQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.655203302 (558h45m24.526286337s) [noise=true , address=145.79.227.14:4838 , piece_id=YSI6KQXM2RUSL7INFCAS5UW3Y7WBEPNR7XZ2HXVVWTBPQEI3PNNQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.746424084 (558h45m24.617507129s) [noise=true , address=45.91.116.10:4083 , piece_id=HC4XDEGOLL5WRTD2M5PEIW3DOA62JDWFVGF3SARAEYFLL2UJH4AA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758832783 (558h45m24.629915828s) [noise=true , address=45.91.116.5:4299 , piece_id=ZIL5KRPIZ7HOLWGUQYSWBPVNNSMYSP3SOJEARHCQBU77XJV7O3OA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: <nil> Dec 28 22:14:24.758860997 (558h45m24.629944032s) [seg_pos={0 0}, upload=489861]: Segment reached optimal threshold of 13 pieces. Dec 28 22:14:24.759139777 (558h45m24.630222823s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759150097 (558h45m24.630233142s) [noise=true , address=77.90.31.15:4697 , piece_id=YGTU3VVURDXCYIYBHFRP256E6R4JGBX3X3VDH7VNOTJSQO3PRO6Q, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759238916 (558h45m24.630321971s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759259204 (558h45m24.63034226s) [noise=true , address=45.91.116.8:4047 , piece_id=BZMURTQLRNC3FIHJQKNX7NWOWTOT74ZXDTPVB2CF7JIPIDP3NJQA, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759466830 (558h45m24.630549875s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 22:14:24.759475446 (558h45m24.630558481s) [noise=true , address=45.91.116.9:4264 , piece_id=GDNH5SBLPB46ZMBKZZCW3LIV7DHIC6BBLFF6SQLT4HX2BLUEXYDQ, seg_pos={0 0}, upload=489861]: Long tail cancellation due to optimal threshold Dec 28 22:14:24.759499041 (558h45m24.630582086s) [seg_pos={0 0}, upload=489861]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 22:14:24.759594203 (558h45m24.630677248s) [upload=489861]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 22:14:24.759601757 (558h45m24.630684812s) [upload=489861]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 22:14:25.061964084 (558h45m24.933047119s) [upload=489861]: Done uploading segments... Dec 28 22:14:25.061984724 (558h45m24.933067759s) [upload=489861]: Upload finished. err: <nil> Duration: 30.002737228s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 21.836462338s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:13:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 21.83649008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.177328177s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:12:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 6.177352924s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.196598532s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:12:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 6.19662846s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.664778532s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:11:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 5.66479841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.096282318s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:11:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 6.096308417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.864698411s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:10:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 5.864723728s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.047852637s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:09:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 6.047878385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.86177926s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:08:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 5.861810469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.959748994s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:07:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 5.959776185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.148213036s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:07:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 6.148240648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.236674034s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:06:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 6.236706135s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.716717344s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:06:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 5.716755196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.00423863s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:05:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 6.004294506s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.832760844s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:04:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 5.832788427s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.980113322s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:04:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 5.980134722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.930550398s trace: https://jaeger.dev.storj.io/trace/601a07d6b82bc6e9 FAIL (2025-12-28 22:03:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 21:37:07.740009777 (558h8m7.611092832s) [upload=489308]: Starting upload Dec 28 21:37:07.740181805 (558h8m7.61126484s) [upload=489308]: Done starting upload Dec 28 21:37:07.740288488 (558h8m7.611371523s) [upload=489308]: Uploading segments... Dec 28 21:37:07.740521131 (558h8m7.611604176s) [upload=489308]: Got next segment. Inline: false Dec 28 21:37:07.740569313 (558h8m7.611652348s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 21:37:07.740575554 (558h8m7.611658589s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 21:37:07.986706010 (558h8m7.857789045s) [seg_pos={0 0}, upload=489308]: Begin upload segment... Dec 28 21:37:07.986768990 (558h8m7.857852025s) [seg_pos={0 0}, upload=489308]: Done begin upload segment. Dec 28 21:37:07.986812643 (558h8m7.857895678s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.986944694 (558h8m7.858027739s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.987759816 (558h8m7.858842861s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989016490 (558h8m7.860099535s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989102032 (558h8m7.860185067s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989452429 (558h8m7.860535474s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.989751720 (558h8m7.860834755s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990027695 (558h8m7.86111073s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990159485 (558h8m7.86124252s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990470037 (558h8m7.861553082s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990660540 (558h8m7.861743585s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.990759328 (558h8m7.861842363s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991226107 (558h8m7.862309142s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.991882276 (558h8m7.862965321s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992237132 (558h8m7.863320177s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:07.992594793 (558h8m7.863677828s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Uploading piece... Dec 28 21:37:09.611123803 (558h8m9.482206848s) [upload=489308]: Next returned nil segment Dec 28 21:37:09.611140486 (558h8m9.482223531s) [upload=489308]: Waiting for error group managing segments... Dec 28 21:37:09.990903640 (558h8m9.861986675s) [noise=true , address=145.79.227.50:4374 , piece_id=ROA33X7LSTSUZCU7YN3PNBLRJXZHDE4JMM3R4AL2422ULZPRFW3Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:09.995717617 (558h8m9.866800652s) [noise=true , address=145.79.227.51:4409 , piece_id=Y4KSTBP5IU43RBPOVQTD5GWAPSMHWSLFGUGC7G45I72A6I3SNISQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.003707571 (558h8m9.874790616s) [noise=true , address=145.79.227.7:4295 , piece_id=IVZICUV6STLNCAGI3IA3FW3YBUDWMN2KR4AWLCOI4EK5IVOHV2BQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.004795953 (558h8m9.875878988s) [noise=true , address=145.79.227.46:4602 , piece_id=62MIDEK7GIH5CEUNTL5OK7MA3RE5DQ6IT2WKLJZ77ERLN4EJGA4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.006489538 (558h8m9.877572573s) [noise=true , address=145.79.227.37:4566 , piece_id=C72B2H4MRS7IJKVEMKWI5G2RWE4W3JAGV6A5G7BD4OR7ZT5QIJRA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.011914669 (558h8m9.882997714s) [noise=true , address=145.79.227.45:4229 , piece_id=AJZX75IR4HPSOOTP5YPXWJUTHTBMYHPXLWBGAWM54ELGZZZNQP4Q, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.013049028 (558h8m9.884132073s) [noise=true , address=145.79.227.11:4731 , piece_id=L5KN7CSAURDZFSOMEMHAITETDF2IWS6MRTQUTU3PAFTTYSNG72VQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.019708570 (558h8m9.890791605s) [noise=true , address=145.79.227.36:4518 , piece_id=OULS2M3NO52345WAT7KTH7ESLX25IHF7RD2VEU2MOPXNT2MIHW4A, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.152290950 (558h8m10.023373985s) [noise=true , address=77.90.31.17:4773 , piece_id=BCC242LTIMG5DUR7R6ZCE3IQ3TPYGHNQAOKBMHHF6T5ETAUIH4CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.173735063 (558h8m10.044818108s) [noise=true , address=77.90.31.19:4844 , piece_id=B5YJWYROQ3ZBF6NOO55ZIZA7WHUZUPSAC24MJGDVUVHHDBEB3SBQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.186556858 (558h8m10.057639903s) [noise=true , address=77.90.31.14:4663 , piece_id=B7JZRUZNU3B2QIDZS775RRXRI6JLII76HUVD4DUVVTGEUENTFVFQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.190856035 (558h8m10.06193908s) [noise=true , address=45.91.116.13:4153 , piece_id=UMPKM5FRBPANILUDZMD66OFGDBA64YAAESS2HGCHH4VZIANPG3CA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196292837 (558h8m10.067375882s) [noise=true , address=45.91.116.7:4009 , piece_id=N5NLEIBEK3YZAJ45AIC5J3D225VCVIDXRFSJUWZKOJVE43WXFBVQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: <nil> Dec 28 21:37:10.196312604 (558h8m10.067395639s) [seg_pos={0 0}, upload=489308]: Segment reached optimal threshold of 13 pieces. Dec 28 21:37:10.196505802 (558h8m10.067588848s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196512184 (558h8m10.06759523s) [noise=true , address=45.91.116.6:4976 , piece_id=2UINTNLVPZJS2FU3UQINUPJGKXIEJM2EJMR3GZ5K4YYLF42WOPIQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196746791 (558h8m10.067829827s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196752693 (558h8m10.067835748s) [noise=true , address=77.90.31.11:4553 , piece_id=5IOLTGZ6UQF5X2V4KMQBAJKW2OBEM2KBD3GEODUFDHTFU4Q2IRAA, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196852012 (558h8m10.067935047s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 21:37:10.196857622 (558h8m10.067940657s) [noise=true , address=45.91.116.12:4227 , piece_id=T4GBNEXQABSTWZSAI2OZE26VODUFH5MDQIYJHZKMBK6KSR7DTMOQ, seg_pos={0 0}, upload=489308]: Long tail cancellation due to optimal threshold Dec 28 21:37:10.196875868 (558h8m10.067958913s) [seg_pos={0 0}, upload=489308]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 21:37:10.196949748 (558h8m10.068032793s) [upload=489308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 21:37:10.196954918 (558h8m10.068037953s) [upload=489308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 21:37:10.493321311 (558h8m10.364404346s) [upload=489308]: Done uploading segments... Dec 28 21:37:10.493341338 (558h8m10.364424374s) [upload=489308]: Upload finished. err: <nil> Duration: 5.930568162s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)