Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............X...........XXX...................XX..X......X.] duration: 5.886850802s trace: https://jaeger.dev.storj.io/trace/5df9cc26463edeeb FAIL (2025-12-25 11:09:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: NSCW3BZS32TWCI3UFAP4K4PWPJEEKJSNDHYY5QNYFVBQMIY4FSUA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 3OFKIXI5N4JCHLYPTZHQDOTB2CNYO6ZCLIX6G2PTEDJYVR3LD6QA) context deadline exceeded ======LOGS====== Dec 25 11:08:54.542987535 (475h39m54.41407058s) [upload=417523]: Starting upload Dec 25 11:08:54.543074060 (475h39m54.414157126s) [upload=417523]: Done starting upload Dec 25 11:08:54.543179922 (475h39m54.414262987s) [upload=417523]: Uploading segments... Dec 25 11:08:54.543339004 (475h39m54.414422039s) [upload=417523]: Got next segment. Inline: false Dec 25 11:08:54.543383570 (475h39m54.414466645s) [upload=417523]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:08:54.543391054 (475h39m54.414474119s) [upload=417523]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:08:54.789912769 (475h39m54.660995824s) [seg_pos={0 0}, upload=417523]: Begin upload segment... Dec 25 11:08:54.789973926 (475h39m54.661056961s) [seg_pos={0 0}, upload=417523]: Done begin upload segment. Dec 25 11:08:54.789996228 (475h39m54.661079263s) [noise=true , address=145.79.227.30:4761 , piece_id=MACTKY4TUSFJUSOC6ZLNLOC5LITXMONJWJNSK5GQX7NJOYCANNOA, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.790022308 (475h39m54.661105353s) [noise=true , address=145.79.227.33:4869 , piece_id=RE4I6XIUBPVLZ27X562VQVS6PVZSEVKSV2S22QFF5F6SBICKGUMQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.790348870 (475h39m54.661431915s) [noise=true , address=145.79.227.52:4447 , piece_id=CI7UUEKFKJKNFGBUNQQWAR7H44LTXV35WAPN22F5C7PBNAXJLJSQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.790860904 (475h39m54.661943949s) [noise=true , address=145.79.227.41:4086 , piece_id=Z55GTZCIWQWDD44AVWS7Q3B7HXCZGYSG3Z5FP5OIDTFGKGL3BZXQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.791209147 (475h39m54.662292182s) [noise=true , address=145.79.227.3:4150 , piece_id=HZZBK7MZN5CFGOF3WZIIAL4LXDX73YOPLMWQ7L5JQDH7YPOAVE4Q, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.791543384 (475h39m54.662626419s) [noise=true , address=145.79.227.38:4978 , piece_id=O4BNAXXD2YZAE6Q2SSIZ6D62L4XFJPJZRYLLZSYOESICAIYU77IA, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.791929960 (475h39m54.663012995s) [noise=true , address=145.79.227.51:4411 , piece_id=CRCISYMJGCNC2IZGSJ46F7GMXQPRZW4FL4IM6LWVWS3ZLCSWSEBA, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.792342997 (475h39m54.663426032s) [noise=true , address=145.79.227.48:4303 , piece_id=ISO3Y2AWT3MEKRB3BYHJCAHZYATXINAGIGXAKVNAG6TPBDAG6ZIQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.792878676 (475h39m54.663961712s) [noise=true , address=45.91.116.13:4155 , piece_id=NSCW3BZS32TWCI3UFAP4K4PWPJEEKJSNDHYY5QNYFVBQMIY4FSUA, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.793995493 (475h39m54.665078538s) [noise=true , address=145.79.227.2:4115 , piece_id=4NJDWAL4CA4EZPP4YFI45SWOGPITV7V6PEAVCEH6A7TJGXKUUKHQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.794351320 (475h39m54.665434365s) [noise=true , address=145.79.227.45:4231 , piece_id=KXOUJ2VPO5XBB5QWY24DPIRM22NSU2GSZBS3U3QECD2RR3VCTICQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.795946809 (475h39m54.667029854s) [noise=true , address=145.79.227.49:4337 , piece_id=HHYQK3O24IZZXMSDHLS5XWHKL47HY7ZUNI6M2PQOXF7KIBZC7SOA, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.798242360 (475h39m54.669325395s) [noise=true , address=45.91.116.4:4939 , piece_id=5SVR7CRHOSRABV7OHRHGKHPIOGYMPQBJZYAQDARN77ADKBMG3KKQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.799850933 (475h39m54.670933968s) [noise=true , address=145.79.227.11:4731 , piece_id=D6JP2V2DK7J4CPE335OVYYMJJLUQNEYG5QXWYLUO7NC43D7CM74Q, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.801480696 (475h39m54.672563741s) [noise=true , address=77.90.31.12:4590 , piece_id=XI236TKWODWETJLEDVMQXMFMILKHHFMBMODZDM55ACI6YDBNKJMQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.802804407 (475h39m54.673887442s) [noise=true , address=77.90.31.19:4844 , piece_id=3OFKIXI5N4JCHLYPTZHQDOTB2CNYO6ZCLIX6G2PTEDJYVR3LD6QA, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:56.512388508 (475h39m56.383471543s) [upload=417523]: Next returned nil segment Dec 25 11:08:56.512417193 (475h39m56.383500228s) [upload=417523]: Waiting for error group managing segments... Dec 25 11:08:56.791370964 (475h39m56.662454009s) [noise=true , address=145.79.227.2:4115 , piece_id=4NJDWAL4CA4EZPP4YFI45SWOGPITV7V6PEAVCEH6A7TJGXKUUKHQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.798971015 (475h39m56.67005406s) [noise=true , address=145.79.227.33:4869 , piece_id=RE4I6XIUBPVLZ27X562VQVS6PVZSEVKSV2S22QFF5F6SBICKGUMQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.805622110 (475h39m56.676705155s) [noise=true , address=145.79.227.3:4150 , piece_id=HZZBK7MZN5CFGOF3WZIIAL4LXDX73YOPLMWQ7L5JQDH7YPOAVE4Q, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.806640118 (475h39m56.677723163s) [noise=true , address=145.79.227.51:4411 , piece_id=CRCISYMJGCNC2IZGSJ46F7GMXQPRZW4FL4IM6LWVWS3ZLCSWSEBA, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.809985829 (475h39m56.681068864s) [noise=true , address=145.79.227.38:4978 , piece_id=O4BNAXXD2YZAE6Q2SSIZ6D62L4XFJPJZRYLLZSYOESICAIYU77IA, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.813039514 (475h39m56.684122549s) [noise=true , address=145.79.227.41:4086 , piece_id=Z55GTZCIWQWDD44AVWS7Q3B7HXCZGYSG3Z5FP5OIDTFGKGL3BZXQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.813840219 (475h39m56.684923254s) [noise=true , address=145.79.227.48:4303 , piece_id=ISO3Y2AWT3MEKRB3BYHJCAHZYATXINAGIGXAKVNAG6TPBDAG6ZIQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.821155207 (475h39m56.692238242s) [noise=true , address=145.79.227.52:4447 , piece_id=CI7UUEKFKJKNFGBUNQQWAR7H44LTXV35WAPN22F5C7PBNAXJLJSQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.821695707 (475h39m56.692778752s) [noise=true , address=145.79.227.45:4231 , piece_id=KXOUJ2VPO5XBB5QWY24DPIRM22NSU2GSZBS3U3QECD2RR3VCTICQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.837091814 (475h39m56.708174849s) [noise=true , address=145.79.227.49:4337 , piece_id=HHYQK3O24IZZXMSDHLS5XWHKL47HY7ZUNI6M2PQOXF7KIBZC7SOA, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.843421736 (475h39m56.714504771s) [noise=true , address=145.79.227.11:4731 , piece_id=D6JP2V2DK7J4CPE335OVYYMJJLUQNEYG5QXWYLUO7NC43D7CM74Q, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:57.035504562 (475h39m56.906587607s) [noise=true , address=45.91.116.13:4155 , piece_id=NSCW3BZS32TWCI3UFAP4K4PWPJEEKJSNDHYY5QNYFVBQMIY4FSUA, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:57.044104227 (475h39m56.915187272s) [noise=true , address=77.90.31.19:4844 , piece_id=3OFKIXI5N4JCHLYPTZHQDOTB2CNYO6ZCLIX6G2PTEDJYVR3LD6QA, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:57.044127361 (475h39m56.915210396s) [seg_pos={0 0}, upload=417523]: Segment reached optimal threshold of 13 pieces. Dec 25 11:08:57.044453653 (475h39m56.915536688s) [noise=true , address=45.91.116.4:4939 , piece_id=5SVR7CRHOSRABV7OHRHGKHPIOGYMPQBJZYAQDARN77ADKBMG3KKQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:57.044471837 (475h39m56.915554882s) [noise=true , address=145.79.227.30:4761 , piece_id=MACTKY4TUSFJUSOC6ZLNLOC5LITXMONJWJNSK5GQX7NJOYCANNOA, seg_pos={0 0}, upload=417523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 11:08:57.044481145 (475h39m56.9155642s) [noise=true , address=145.79.227.30:4761 , piece_id=MACTKY4TUSFJUSOC6ZLNLOC5LITXMONJWJNSK5GQX7NJOYCANNOA, seg_pos={0 0}, upload=417523]: Long tail cancellation due to optimal threshold Dec 25 11:08:57.044637272 (475h39m56.915720317s) [noise=true , address=77.90.31.12:4590 , piece_id=XI236TKWODWETJLEDVMQXMFMILKHHFMBMODZDM55ACI6YDBNKJMQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:35162->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:35162->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35162->77.90.31.12:4590: use of closed network connection Dec 25 11:08:57.044645437 (475h39m56.915728482s) [noise=true , address=77.90.31.12:4590 , piece_id=XI236TKWODWETJLEDVMQXMFMILKHHFMBMODZDM55ACI6YDBNKJMQ, seg_pos={0 0}, upload=417523]: Long tail cancellation due to optimal threshold Dec 25 11:08:57.044666398 (475h39m56.915749433s) [seg_pos={0 0}, upload=417523]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 25 11:08:57.044749916 (475h39m56.915832951s) [upload=417523]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:08:57.044755677 (475h39m56.915838722s) [upload=417523]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:08:57.332673917 (475h39m57.203756962s) [upload=417523]: Done uploading segments... Dec 25 11:08:57.332697132 (475h39m57.203780177s) [upload=417523]: Upload finished. err: <nil> Duration: 5.886874187s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............X...........XXX...................XX..X......X] duration: 30.003871671s trace: https://jaeger.dev.storj.io/trace/5df9cc26463edeeb FAIL (2025-12-25 11:09:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: NSCW3BZS32TWCI3UFAP4K4PWPJEEKJSNDHYY5QNYFVBQMIY4FSUA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 3OFKIXI5N4JCHLYPTZHQDOTB2CNYO6ZCLIX6G2PTEDJYVR3LD6QA) context deadline exceeded ======LOGS====== Dec 25 11:08:54.542987535 (475h39m54.41407058s) [upload=417523]: Starting upload Dec 25 11:08:54.543074060 (475h39m54.414157126s) [upload=417523]: Done starting upload Dec 25 11:08:54.543179922 (475h39m54.414262987s) [upload=417523]: Uploading segments... Dec 25 11:08:54.543339004 (475h39m54.414422039s) [upload=417523]: Got next segment. Inline: false Dec 25 11:08:54.543383570 (475h39m54.414466645s) [upload=417523]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:08:54.543391054 (475h39m54.414474119s) [upload=417523]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:08:54.789912769 (475h39m54.660995824s) [seg_pos={0 0}, upload=417523]: Begin upload segment... Dec 25 11:08:54.789973926 (475h39m54.661056961s) [seg_pos={0 0}, upload=417523]: Done begin upload segment. Dec 25 11:08:54.789996228 (475h39m54.661079263s) [noise=true , address=145.79.227.30:4761 , piece_id=MACTKY4TUSFJUSOC6ZLNLOC5LITXMONJWJNSK5GQX7NJOYCANNOA, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.790022308 (475h39m54.661105353s) [noise=true , address=145.79.227.33:4869 , piece_id=RE4I6XIUBPVLZ27X562VQVS6PVZSEVKSV2S22QFF5F6SBICKGUMQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.790348870 (475h39m54.661431915s) [noise=true , address=145.79.227.52:4447 , piece_id=CI7UUEKFKJKNFGBUNQQWAR7H44LTXV35WAPN22F5C7PBNAXJLJSQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.790860904 (475h39m54.661943949s) [noise=true , address=145.79.227.41:4086 , piece_id=Z55GTZCIWQWDD44AVWS7Q3B7HXCZGYSG3Z5FP5OIDTFGKGL3BZXQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.791209147 (475h39m54.662292182s) [noise=true , address=145.79.227.3:4150 , piece_id=HZZBK7MZN5CFGOF3WZIIAL4LXDX73YOPLMWQ7L5JQDH7YPOAVE4Q, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.791543384 (475h39m54.662626419s) [noise=true , address=145.79.227.38:4978 , piece_id=O4BNAXXD2YZAE6Q2SSIZ6D62L4XFJPJZRYLLZSYOESICAIYU77IA, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.791929960 (475h39m54.663012995s) [noise=true , address=145.79.227.51:4411 , piece_id=CRCISYMJGCNC2IZGSJ46F7GMXQPRZW4FL4IM6LWVWS3ZLCSWSEBA, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.792342997 (475h39m54.663426032s) [noise=true , address=145.79.227.48:4303 , piece_id=ISO3Y2AWT3MEKRB3BYHJCAHZYATXINAGIGXAKVNAG6TPBDAG6ZIQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.792878676 (475h39m54.663961712s) [noise=true , address=45.91.116.13:4155 , piece_id=NSCW3BZS32TWCI3UFAP4K4PWPJEEKJSNDHYY5QNYFVBQMIY4FSUA, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.793995493 (475h39m54.665078538s) [noise=true , address=145.79.227.2:4115 , piece_id=4NJDWAL4CA4EZPP4YFI45SWOGPITV7V6PEAVCEH6A7TJGXKUUKHQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.794351320 (475h39m54.665434365s) [noise=true , address=145.79.227.45:4231 , piece_id=KXOUJ2VPO5XBB5QWY24DPIRM22NSU2GSZBS3U3QECD2RR3VCTICQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.795946809 (475h39m54.667029854s) [noise=true , address=145.79.227.49:4337 , piece_id=HHYQK3O24IZZXMSDHLS5XWHKL47HY7ZUNI6M2PQOXF7KIBZC7SOA, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.798242360 (475h39m54.669325395s) [noise=true , address=45.91.116.4:4939 , piece_id=5SVR7CRHOSRABV7OHRHGKHPIOGYMPQBJZYAQDARN77ADKBMG3KKQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.799850933 (475h39m54.670933968s) [noise=true , address=145.79.227.11:4731 , piece_id=D6JP2V2DK7J4CPE335OVYYMJJLUQNEYG5QXWYLUO7NC43D7CM74Q, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.801480696 (475h39m54.672563741s) [noise=true , address=77.90.31.12:4590 , piece_id=XI236TKWODWETJLEDVMQXMFMILKHHFMBMODZDM55ACI6YDBNKJMQ, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:54.802804407 (475h39m54.673887442s) [noise=true , address=77.90.31.19:4844 , piece_id=3OFKIXI5N4JCHLYPTZHQDOTB2CNYO6ZCLIX6G2PTEDJYVR3LD6QA, seg_pos={0 0}, upload=417523]: Uploading piece... Dec 25 11:08:56.512388508 (475h39m56.383471543s) [upload=417523]: Next returned nil segment Dec 25 11:08:56.512417193 (475h39m56.383500228s) [upload=417523]: Waiting for error group managing segments... Dec 25 11:08:56.791370964 (475h39m56.662454009s) [noise=true , address=145.79.227.2:4115 , piece_id=4NJDWAL4CA4EZPP4YFI45SWOGPITV7V6PEAVCEH6A7TJGXKUUKHQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.798971015 (475h39m56.67005406s) [noise=true , address=145.79.227.33:4869 , piece_id=RE4I6XIUBPVLZ27X562VQVS6PVZSEVKSV2S22QFF5F6SBICKGUMQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.805622110 (475h39m56.676705155s) [noise=true , address=145.79.227.3:4150 , piece_id=HZZBK7MZN5CFGOF3WZIIAL4LXDX73YOPLMWQ7L5JQDH7YPOAVE4Q, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.806640118 (475h39m56.677723163s) [noise=true , address=145.79.227.51:4411 , piece_id=CRCISYMJGCNC2IZGSJ46F7GMXQPRZW4FL4IM6LWVWS3ZLCSWSEBA, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.809985829 (475h39m56.681068864s) [noise=true , address=145.79.227.38:4978 , piece_id=O4BNAXXD2YZAE6Q2SSIZ6D62L4XFJPJZRYLLZSYOESICAIYU77IA, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.813039514 (475h39m56.684122549s) [noise=true , address=145.79.227.41:4086 , piece_id=Z55GTZCIWQWDD44AVWS7Q3B7HXCZGYSG3Z5FP5OIDTFGKGL3BZXQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.813840219 (475h39m56.684923254s) [noise=true , address=145.79.227.48:4303 , piece_id=ISO3Y2AWT3MEKRB3BYHJCAHZYATXINAGIGXAKVNAG6TPBDAG6ZIQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.821155207 (475h39m56.692238242s) [noise=true , address=145.79.227.52:4447 , piece_id=CI7UUEKFKJKNFGBUNQQWAR7H44LTXV35WAPN22F5C7PBNAXJLJSQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.821695707 (475h39m56.692778752s) [noise=true , address=145.79.227.45:4231 , piece_id=KXOUJ2VPO5XBB5QWY24DPIRM22NSU2GSZBS3U3QECD2RR3VCTICQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.837091814 (475h39m56.708174849s) [noise=true , address=145.79.227.49:4337 , piece_id=HHYQK3O24IZZXMSDHLS5XWHKL47HY7ZUNI6M2PQOXF7KIBZC7SOA, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:56.843421736 (475h39m56.714504771s) [noise=true , address=145.79.227.11:4731 , piece_id=D6JP2V2DK7J4CPE335OVYYMJJLUQNEYG5QXWYLUO7NC43D7CM74Q, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:57.035504562 (475h39m56.906587607s) [noise=true , address=45.91.116.13:4155 , piece_id=NSCW3BZS32TWCI3UFAP4K4PWPJEEKJSNDHYY5QNYFVBQMIY4FSUA, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:57.044104227 (475h39m56.915187272s) [noise=true , address=77.90.31.19:4844 , piece_id=3OFKIXI5N4JCHLYPTZHQDOTB2CNYO6ZCLIX6G2PTEDJYVR3LD6QA, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:57.044127361 (475h39m56.915210396s) [seg_pos={0 0}, upload=417523]: Segment reached optimal threshold of 13 pieces. Dec 25 11:08:57.044453653 (475h39m56.915536688s) [noise=true , address=45.91.116.4:4939 , piece_id=5SVR7CRHOSRABV7OHRHGKHPIOGYMPQBJZYAQDARN77ADKBMG3KKQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: <nil> Dec 25 11:08:57.044471837 (475h39m56.915554882s) [noise=true , address=145.79.227.30:4761 , piece_id=MACTKY4TUSFJUSOC6ZLNLOC5LITXMONJWJNSK5GQX7NJOYCANNOA, seg_pos={0 0}, upload=417523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 11:08:57.044481145 (475h39m56.9155642s) [noise=true , address=145.79.227.30:4761 , piece_id=MACTKY4TUSFJUSOC6ZLNLOC5LITXMONJWJNSK5GQX7NJOYCANNOA, seg_pos={0 0}, upload=417523]: Long tail cancellation due to optimal threshold Dec 25 11:08:57.044637272 (475h39m56.915720317s) [noise=true , address=77.90.31.12:4590 , piece_id=XI236TKWODWETJLEDVMQXMFMILKHHFMBMODZDM55ACI6YDBNKJMQ, seg_pos={0 0}, upload=417523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:35162->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:35162->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35162->77.90.31.12:4590: use of closed network connection Dec 25 11:08:57.044645437 (475h39m56.915728482s) [noise=true , address=77.90.31.12:4590 , piece_id=XI236TKWODWETJLEDVMQXMFMILKHHFMBMODZDM55ACI6YDBNKJMQ, seg_pos={0 0}, upload=417523]: Long tail cancellation due to optimal threshold Dec 25 11:08:57.044666398 (475h39m56.915749433s) [seg_pos={0 0}, upload=417523]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 25 11:08:57.044749916 (475h39m56.915832951s) [upload=417523]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:08:57.044755677 (475h39m56.915838722s) [upload=417523]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:08:57.332673917 (475h39m57.203756962s) [upload=417523]: Done uploading segments... Dec 25 11:08:57.332697132 (475h39m57.203780177s) [upload=417523]: Upload finished. err: <nil> Duration: 30.003894485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............X...........XXX...................XX..X......] duration: 5.694844814s trace: https://jaeger.dev.storj.io/trace/2dc5d425884c83bb FAIL (2025-12-25 11:08:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 11:03:59.515608956 (475h34m59.386692001s) [upload=417450]: Starting upload Dec 25 11:03:59.515687064 (475h34m59.386770119s) [upload=417450]: Done starting upload Dec 25 11:03:59.515805179 (475h34m59.386888224s) [upload=417450]: Uploading segments... Dec 25 11:03:59.515923294 (475h34m59.387006329s) [upload=417450]: Got next segment. Inline: false Dec 25 11:03:59.515992456 (475h34m59.387075491s) [upload=417450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:03:59.515997395 (475h34m59.38708043s) [upload=417450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:03:59.763382274 (475h34m59.634465309s) [seg_pos={0 0}, upload=417450]: Begin upload segment... Dec 25 11:03:59.763426679 (475h34m59.634509724s) [seg_pos={0 0}, upload=417450]: Done begin upload segment. Dec 25 11:03:59.763452508 (475h34m59.634535543s) [noise=true , address=145.79.227.37:4567 , piece_id=PV7WZX2MYWKBW6SC5R6XKJEKMECQ7E45IWI3WGCN2UJCYRMQ3BBA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.763553290 (475h34m59.634636325s) [noise=true , address=145.79.227.52:4447 , piece_id=NGMJBLCAJ5OKYWLT5Q7MG3C6NRU7WZZQVDOB77BJH6J5GX6NO6KQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764047371 (475h34m59.635130406s) [noise=true , address=45.91.116.4:4937 , piece_id=64AINFG4GLFSYFIGEZ4JXDD2EECEAFRENLILEARTFS3L4WNBDISQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764408799 (475h34m59.635491834s) [noise=true , address=145.79.227.47:4266 , piece_id=42YSOQM5OG6FPIJ4LLFLNYGCCAT4X6YNDA4IH45AZDQFLZDXW5PQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764673713 (475h34m59.635756748s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764692268 (475h34m59.635775303s) [noise=true , address=145.79.227.49:4338 , piece_id=KEDFC64SQTVWKRANH6PBRB4UIHULTTO2JLHXDZDUZHTAGYVSI7QQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764969907 (475h34m59.636052942s) [noise=true , address=77.90.31.21:4880 , piece_id=3UNC5SGWF6K5PZV24W5ZR6E7VA4XTP4Z3L64RH5ZDK6LITGDRBOQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765257254 (475h34m59.636340289s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765284636 (475h34m59.636367671s) [noise=true , address=145.79.227.6:4258 , piece_id=S6GJBYDLOGHYMH234TEX3ZZ7QYPHPKAGUWZLF3VDX3Q7DQNCEBQQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765664510 (475h34m59.636747555s) [noise=true , address=145.79.227.4:4187 , piece_id=C4A5R2WGU2D7MCLK2ZITZKAX4JA37C6V63JQLRZ6XZ3PYRVMRUUA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765678806 (475h34m59.636761841s) [noise=true , address=145.79.227.38:4977 , piece_id=NS6W2YFFCWZCTS74C2NGJTFJIYEMP55L2SBUVDB2ZDUL2EM2WT4A, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765969811 (475h34m59.637052846s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766154823 (475h34m59.637237868s) [noise=true , address=145.79.227.7:4294 , piece_id=VB2I6ORRUF5KFQZDI2NMUP75JAA4VH5EX5IAW74WYRICLAEZ57ZA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766554695 (475h34m59.63763773s) [noise=true , address=77.90.31.13:4627 , piece_id=5JFHJ4HUVYR27UTCPBXMPARMLXSMYOSTTYTI6FGEBI4KDTT7L7WA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766958664 (475h34m59.638041699s) [noise=true , address=145.79.227.42:4121 , piece_id=HUXHLCXPGXU6LGZBVT5B2GIMLHFJDYAV3HDPJKXM3RPT5OS7BGIQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.767318640 (475h34m59.638401675s) [noise=true , address=45.91.116.14:4191 , piece_id=XA53IWCSYI2VKZZPX7FWDZM5N5CXCWPCCA7F2NHML6MTBR2MSZYQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:04:01.388268928 (475h35m1.259351973s) [upload=417450]: Next returned nil segment Dec 25 11:04:01.388286351 (475h35m1.259369406s) [upload=417450]: Waiting for error group managing segments... Dec 25 11:04:01.773288615 (475h35m1.64437166s) [noise=true , address=145.79.227.37:4567 , piece_id=PV7WZX2MYWKBW6SC5R6XKJEKMECQ7E45IWI3WGCN2UJCYRMQ3BBA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.774294370 (475h35m1.645377415s) [noise=true , address=145.79.227.38:4977 , piece_id=NS6W2YFFCWZCTS74C2NGJTFJIYEMP55L2SBUVDB2ZDUL2EM2WT4A, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.777947667 (475h35m1.649030712s) [noise=true , address=145.79.227.42:4121 , piece_id=HUXHLCXPGXU6LGZBVT5B2GIMLHFJDYAV3HDPJKXM3RPT5OS7BGIQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.783365162 (475h35m1.654448207s) [noise=true , address=145.79.227.52:4447 , piece_id=NGMJBLCAJ5OKYWLT5Q7MG3C6NRU7WZZQVDOB77BJH6J5GX6NO6KQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.783464501 (475h35m1.654547546s) [noise=true , address=145.79.227.47:4266 , piece_id=42YSOQM5OG6FPIJ4LLFLNYGCCAT4X6YNDA4IH45AZDQFLZDXW5PQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.784244136 (475h35m1.655327181s) [noise=true , address=145.79.227.49:4338 , piece_id=KEDFC64SQTVWKRANH6PBRB4UIHULTTO2JLHXDZDUZHTAGYVSI7QQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.790940567 (475h35m1.662023612s) [noise=true , address=145.79.227.7:4294 , piece_id=VB2I6ORRUF5KFQZDI2NMUP75JAA4VH5EX5IAW74WYRICLAEZ57ZA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.799823582 (475h35m1.670906627s) [noise=true , address=145.79.227.6:4258 , piece_id=S6GJBYDLOGHYMH234TEX3ZZ7QYPHPKAGUWZLF3VDX3Q7DQNCEBQQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.830799615 (475h35m1.70188266s) [noise=true , address=145.79.227.4:4187 , piece_id=C4A5R2WGU2D7MCLK2ZITZKAX4JA37C6V63JQLRZ6XZ3PYRVMRUUA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.016322589 (475h35m1.887405634s) [noise=true , address=77.90.31.13:4627 , piece_id=5JFHJ4HUVYR27UTCPBXMPARMLXSMYOSTTYTI6FGEBI4KDTT7L7WA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.041203226 (475h35m1.912286261s) [noise=true , address=45.91.116.4:4937 , piece_id=64AINFG4GLFSYFIGEZ4JXDD2EECEAFRENLILEARTFS3L4WNBDISQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.052992725 (475h35m1.92407577s) [noise=true , address=45.91.116.14:4191 , piece_id=XA53IWCSYI2VKZZPX7FWDZM5N5CXCWPCCA7F2NHML6MTBR2MSZYQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.072943965 (475h35m1.94402702s) [noise=true , address=77.90.31.21:4880 , piece_id=3UNC5SGWF6K5PZV24W5ZR6E7VA4XTP4Z3L64RH5ZDK6LITGDRBOQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.072967100 (475h35m1.944050155s) [seg_pos={0 0}, upload=417450]: Segment reached optimal threshold of 13 pieces. Dec 25 11:04:02.073232575 (475h35m1.94431562s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073244077 (475h35m1.944327122s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073394233 (475h35m1.944477288s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073405784 (475h35m1.94448884s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073773144 (475h35m1.944856199s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073784846 (475h35m1.944867891s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073924402 (475h35m1.945007447s) [seg_pos={0 0}, upload=417450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:04:02.074031245 (475h35m1.945114291s) [upload=417450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:04:02.074038449 (475h35m1.945121494s) [upload=417450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:04:02.360149859 (475h35m2.231232915s) [upload=417450]: Done uploading segments... Dec 25 11:04:02.360212960 (475h35m2.231296015s) [upload=417450]: Upload finished. err: <nil> Duration: 5.694874721s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [................X...........XXX...................XX..X.....] duration: 5.721178856s trace: https://jaeger.dev.storj.io/trace/2dc5d425884c83bb FAIL (2025-12-25 11:08:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 11:03:59.515608956 (475h34m59.386692001s) [upload=417450]: Starting upload Dec 25 11:03:59.515687064 (475h34m59.386770119s) [upload=417450]: Done starting upload Dec 25 11:03:59.515805179 (475h34m59.386888224s) [upload=417450]: Uploading segments... Dec 25 11:03:59.515923294 (475h34m59.387006329s) [upload=417450]: Got next segment. Inline: false Dec 25 11:03:59.515992456 (475h34m59.387075491s) [upload=417450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:03:59.515997395 (475h34m59.38708043s) [upload=417450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:03:59.763382274 (475h34m59.634465309s) [seg_pos={0 0}, upload=417450]: Begin upload segment... Dec 25 11:03:59.763426679 (475h34m59.634509724s) [seg_pos={0 0}, upload=417450]: Done begin upload segment. Dec 25 11:03:59.763452508 (475h34m59.634535543s) [noise=true , address=145.79.227.37:4567 , piece_id=PV7WZX2MYWKBW6SC5R6XKJEKMECQ7E45IWI3WGCN2UJCYRMQ3BBA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.763553290 (475h34m59.634636325s) [noise=true , address=145.79.227.52:4447 , piece_id=NGMJBLCAJ5OKYWLT5Q7MG3C6NRU7WZZQVDOB77BJH6J5GX6NO6KQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764047371 (475h34m59.635130406s) [noise=true , address=45.91.116.4:4937 , piece_id=64AINFG4GLFSYFIGEZ4JXDD2EECEAFRENLILEARTFS3L4WNBDISQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764408799 (475h34m59.635491834s) [noise=true , address=145.79.227.47:4266 , piece_id=42YSOQM5OG6FPIJ4LLFLNYGCCAT4X6YNDA4IH45AZDQFLZDXW5PQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764673713 (475h34m59.635756748s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764692268 (475h34m59.635775303s) [noise=true , address=145.79.227.49:4338 , piece_id=KEDFC64SQTVWKRANH6PBRB4UIHULTTO2JLHXDZDUZHTAGYVSI7QQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764969907 (475h34m59.636052942s) [noise=true , address=77.90.31.21:4880 , piece_id=3UNC5SGWF6K5PZV24W5ZR6E7VA4XTP4Z3L64RH5ZDK6LITGDRBOQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765257254 (475h34m59.636340289s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765284636 (475h34m59.636367671s) [noise=true , address=145.79.227.6:4258 , piece_id=S6GJBYDLOGHYMH234TEX3ZZ7QYPHPKAGUWZLF3VDX3Q7DQNCEBQQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765664510 (475h34m59.636747555s) [noise=true , address=145.79.227.4:4187 , piece_id=C4A5R2WGU2D7MCLK2ZITZKAX4JA37C6V63JQLRZ6XZ3PYRVMRUUA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765678806 (475h34m59.636761841s) [noise=true , address=145.79.227.38:4977 , piece_id=NS6W2YFFCWZCTS74C2NGJTFJIYEMP55L2SBUVDB2ZDUL2EM2WT4A, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765969811 (475h34m59.637052846s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766154823 (475h34m59.637237868s) [noise=true , address=145.79.227.7:4294 , piece_id=VB2I6ORRUF5KFQZDI2NMUP75JAA4VH5EX5IAW74WYRICLAEZ57ZA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766554695 (475h34m59.63763773s) [noise=true , address=77.90.31.13:4627 , piece_id=5JFHJ4HUVYR27UTCPBXMPARMLXSMYOSTTYTI6FGEBI4KDTT7L7WA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766958664 (475h34m59.638041699s) [noise=true , address=145.79.227.42:4121 , piece_id=HUXHLCXPGXU6LGZBVT5B2GIMLHFJDYAV3HDPJKXM3RPT5OS7BGIQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.767318640 (475h34m59.638401675s) [noise=true , address=45.91.116.14:4191 , piece_id=XA53IWCSYI2VKZZPX7FWDZM5N5CXCWPCCA7F2NHML6MTBR2MSZYQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:04:01.388268928 (475h35m1.259351973s) [upload=417450]: Next returned nil segment Dec 25 11:04:01.388286351 (475h35m1.259369406s) [upload=417450]: Waiting for error group managing segments... Dec 25 11:04:01.773288615 (475h35m1.64437166s) [noise=true , address=145.79.227.37:4567 , piece_id=PV7WZX2MYWKBW6SC5R6XKJEKMECQ7E45IWI3WGCN2UJCYRMQ3BBA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.774294370 (475h35m1.645377415s) [noise=true , address=145.79.227.38:4977 , piece_id=NS6W2YFFCWZCTS74C2NGJTFJIYEMP55L2SBUVDB2ZDUL2EM2WT4A, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.777947667 (475h35m1.649030712s) [noise=true , address=145.79.227.42:4121 , piece_id=HUXHLCXPGXU6LGZBVT5B2GIMLHFJDYAV3HDPJKXM3RPT5OS7BGIQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.783365162 (475h35m1.654448207s) [noise=true , address=145.79.227.52:4447 , piece_id=NGMJBLCAJ5OKYWLT5Q7MG3C6NRU7WZZQVDOB77BJH6J5GX6NO6KQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.783464501 (475h35m1.654547546s) [noise=true , address=145.79.227.47:4266 , piece_id=42YSOQM5OG6FPIJ4LLFLNYGCCAT4X6YNDA4IH45AZDQFLZDXW5PQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.784244136 (475h35m1.655327181s) [noise=true , address=145.79.227.49:4338 , piece_id=KEDFC64SQTVWKRANH6PBRB4UIHULTTO2JLHXDZDUZHTAGYVSI7QQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.790940567 (475h35m1.662023612s) [noise=true , address=145.79.227.7:4294 , piece_id=VB2I6ORRUF5KFQZDI2NMUP75JAA4VH5EX5IAW74WYRICLAEZ57ZA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.799823582 (475h35m1.670906627s) [noise=true , address=145.79.227.6:4258 , piece_id=S6GJBYDLOGHYMH234TEX3ZZ7QYPHPKAGUWZLF3VDX3Q7DQNCEBQQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.830799615 (475h35m1.70188266s) [noise=true , address=145.79.227.4:4187 , piece_id=C4A5R2WGU2D7MCLK2ZITZKAX4JA37C6V63JQLRZ6XZ3PYRVMRUUA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.016322589 (475h35m1.887405634s) [noise=true , address=77.90.31.13:4627 , piece_id=5JFHJ4HUVYR27UTCPBXMPARMLXSMYOSTTYTI6FGEBI4KDTT7L7WA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.041203226 (475h35m1.912286261s) [noise=true , address=45.91.116.4:4937 , piece_id=64AINFG4GLFSYFIGEZ4JXDD2EECEAFRENLILEARTFS3L4WNBDISQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.052992725 (475h35m1.92407577s) [noise=true , address=45.91.116.14:4191 , piece_id=XA53IWCSYI2VKZZPX7FWDZM5N5CXCWPCCA7F2NHML6MTBR2MSZYQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.072943965 (475h35m1.94402702s) [noise=true , address=77.90.31.21:4880 , piece_id=3UNC5SGWF6K5PZV24W5ZR6E7VA4XTP4Z3L64RH5ZDK6LITGDRBOQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.072967100 (475h35m1.944050155s) [seg_pos={0 0}, upload=417450]: Segment reached optimal threshold of 13 pieces. Dec 25 11:04:02.073232575 (475h35m1.94431562s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073244077 (475h35m1.944327122s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073394233 (475h35m1.944477288s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073405784 (475h35m1.94448884s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073773144 (475h35m1.944856199s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073784846 (475h35m1.944867891s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073924402 (475h35m1.945007447s) [seg_pos={0 0}, upload=417450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:04:02.074031245 (475h35m1.945114291s) [upload=417450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:04:02.074038449 (475h35m1.945121494s) [upload=417450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:04:02.360149859 (475h35m2.231232915s) [upload=417450]: Done uploading segments... Dec 25 11:04:02.360212960 (475h35m2.231296015s) [upload=417450]: Upload finished. err: <nil> Duration: 5.721211558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................X...........XXX...................XX..X....] duration: 6.765550743s trace: https://jaeger.dev.storj.io/trace/2dc5d425884c83bb FAIL (2025-12-25 11:07:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 11:03:59.515608956 (475h34m59.386692001s) [upload=417450]: Starting upload Dec 25 11:03:59.515687064 (475h34m59.386770119s) [upload=417450]: Done starting upload Dec 25 11:03:59.515805179 (475h34m59.386888224s) [upload=417450]: Uploading segments... Dec 25 11:03:59.515923294 (475h34m59.387006329s) [upload=417450]: Got next segment. Inline: false Dec 25 11:03:59.515992456 (475h34m59.387075491s) [upload=417450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:03:59.515997395 (475h34m59.38708043s) [upload=417450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:03:59.763382274 (475h34m59.634465309s) [seg_pos={0 0}, upload=417450]: Begin upload segment... Dec 25 11:03:59.763426679 (475h34m59.634509724s) [seg_pos={0 0}, upload=417450]: Done begin upload segment. Dec 25 11:03:59.763452508 (475h34m59.634535543s) [noise=true , address=145.79.227.37:4567 , piece_id=PV7WZX2MYWKBW6SC5R6XKJEKMECQ7E45IWI3WGCN2UJCYRMQ3BBA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.763553290 (475h34m59.634636325s) [noise=true , address=145.79.227.52:4447 , piece_id=NGMJBLCAJ5OKYWLT5Q7MG3C6NRU7WZZQVDOB77BJH6J5GX6NO6KQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764047371 (475h34m59.635130406s) [noise=true , address=45.91.116.4:4937 , piece_id=64AINFG4GLFSYFIGEZ4JXDD2EECEAFRENLILEARTFS3L4WNBDISQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764408799 (475h34m59.635491834s) [noise=true , address=145.79.227.47:4266 , piece_id=42YSOQM5OG6FPIJ4LLFLNYGCCAT4X6YNDA4IH45AZDQFLZDXW5PQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764673713 (475h34m59.635756748s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764692268 (475h34m59.635775303s) [noise=true , address=145.79.227.49:4338 , piece_id=KEDFC64SQTVWKRANH6PBRB4UIHULTTO2JLHXDZDUZHTAGYVSI7QQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764969907 (475h34m59.636052942s) [noise=true , address=77.90.31.21:4880 , piece_id=3UNC5SGWF6K5PZV24W5ZR6E7VA4XTP4Z3L64RH5ZDK6LITGDRBOQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765257254 (475h34m59.636340289s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765284636 (475h34m59.636367671s) [noise=true , address=145.79.227.6:4258 , piece_id=S6GJBYDLOGHYMH234TEX3ZZ7QYPHPKAGUWZLF3VDX3Q7DQNCEBQQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765664510 (475h34m59.636747555s) [noise=true , address=145.79.227.4:4187 , piece_id=C4A5R2WGU2D7MCLK2ZITZKAX4JA37C6V63JQLRZ6XZ3PYRVMRUUA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765678806 (475h34m59.636761841s) [noise=true , address=145.79.227.38:4977 , piece_id=NS6W2YFFCWZCTS74C2NGJTFJIYEMP55L2SBUVDB2ZDUL2EM2WT4A, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765969811 (475h34m59.637052846s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766154823 (475h34m59.637237868s) [noise=true , address=145.79.227.7:4294 , piece_id=VB2I6ORRUF5KFQZDI2NMUP75JAA4VH5EX5IAW74WYRICLAEZ57ZA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766554695 (475h34m59.63763773s) [noise=true , address=77.90.31.13:4627 , piece_id=5JFHJ4HUVYR27UTCPBXMPARMLXSMYOSTTYTI6FGEBI4KDTT7L7WA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766958664 (475h34m59.638041699s) [noise=true , address=145.79.227.42:4121 , piece_id=HUXHLCXPGXU6LGZBVT5B2GIMLHFJDYAV3HDPJKXM3RPT5OS7BGIQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.767318640 (475h34m59.638401675s) [noise=true , address=45.91.116.14:4191 , piece_id=XA53IWCSYI2VKZZPX7FWDZM5N5CXCWPCCA7F2NHML6MTBR2MSZYQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:04:01.388268928 (475h35m1.259351973s) [upload=417450]: Next returned nil segment Dec 25 11:04:01.388286351 (475h35m1.259369406s) [upload=417450]: Waiting for error group managing segments... Dec 25 11:04:01.773288615 (475h35m1.64437166s) [noise=true , address=145.79.227.37:4567 , piece_id=PV7WZX2MYWKBW6SC5R6XKJEKMECQ7E45IWI3WGCN2UJCYRMQ3BBA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.774294370 (475h35m1.645377415s) [noise=true , address=145.79.227.38:4977 , piece_id=NS6W2YFFCWZCTS74C2NGJTFJIYEMP55L2SBUVDB2ZDUL2EM2WT4A, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.777947667 (475h35m1.649030712s) [noise=true , address=145.79.227.42:4121 , piece_id=HUXHLCXPGXU6LGZBVT5B2GIMLHFJDYAV3HDPJKXM3RPT5OS7BGIQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.783365162 (475h35m1.654448207s) [noise=true , address=145.79.227.52:4447 , piece_id=NGMJBLCAJ5OKYWLT5Q7MG3C6NRU7WZZQVDOB77BJH6J5GX6NO6KQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.783464501 (475h35m1.654547546s) [noise=true , address=145.79.227.47:4266 , piece_id=42YSOQM5OG6FPIJ4LLFLNYGCCAT4X6YNDA4IH45AZDQFLZDXW5PQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.784244136 (475h35m1.655327181s) [noise=true , address=145.79.227.49:4338 , piece_id=KEDFC64SQTVWKRANH6PBRB4UIHULTTO2JLHXDZDUZHTAGYVSI7QQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.790940567 (475h35m1.662023612s) [noise=true , address=145.79.227.7:4294 , piece_id=VB2I6ORRUF5KFQZDI2NMUP75JAA4VH5EX5IAW74WYRICLAEZ57ZA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.799823582 (475h35m1.670906627s) [noise=true , address=145.79.227.6:4258 , piece_id=S6GJBYDLOGHYMH234TEX3ZZ7QYPHPKAGUWZLF3VDX3Q7DQNCEBQQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.830799615 (475h35m1.70188266s) [noise=true , address=145.79.227.4:4187 , piece_id=C4A5R2WGU2D7MCLK2ZITZKAX4JA37C6V63JQLRZ6XZ3PYRVMRUUA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.016322589 (475h35m1.887405634s) [noise=true , address=77.90.31.13:4627 , piece_id=5JFHJ4HUVYR27UTCPBXMPARMLXSMYOSTTYTI6FGEBI4KDTT7L7WA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.041203226 (475h35m1.912286261s) [noise=true , address=45.91.116.4:4937 , piece_id=64AINFG4GLFSYFIGEZ4JXDD2EECEAFRENLILEARTFS3L4WNBDISQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.052992725 (475h35m1.92407577s) [noise=true , address=45.91.116.14:4191 , piece_id=XA53IWCSYI2VKZZPX7FWDZM5N5CXCWPCCA7F2NHML6MTBR2MSZYQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.072943965 (475h35m1.94402702s) [noise=true , address=77.90.31.21:4880 , piece_id=3UNC5SGWF6K5PZV24W5ZR6E7VA4XTP4Z3L64RH5ZDK6LITGDRBOQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.072967100 (475h35m1.944050155s) [seg_pos={0 0}, upload=417450]: Segment reached optimal threshold of 13 pieces. Dec 25 11:04:02.073232575 (475h35m1.94431562s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073244077 (475h35m1.944327122s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073394233 (475h35m1.944477288s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073405784 (475h35m1.94448884s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073773144 (475h35m1.944856199s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073784846 (475h35m1.944867891s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073924402 (475h35m1.945007447s) [seg_pos={0 0}, upload=417450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:04:02.074031245 (475h35m1.945114291s) [upload=417450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:04:02.074038449 (475h35m1.945121494s) [upload=417450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:04:02.360149859 (475h35m2.231232915s) [upload=417450]: Done uploading segments... Dec 25 11:04:02.360212960 (475h35m2.231296015s) [upload=417450]: Upload finished. err: <nil> Duration: 6.765579849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................X...........XXX...................XX..X...] duration: 5.776146244s trace: https://jaeger.dev.storj.io/trace/2dc5d425884c83bb FAIL (2025-12-25 11:06:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 11:03:59.515608956 (475h34m59.386692001s) [upload=417450]: Starting upload Dec 25 11:03:59.515687064 (475h34m59.386770119s) [upload=417450]: Done starting upload Dec 25 11:03:59.515805179 (475h34m59.386888224s) [upload=417450]: Uploading segments... Dec 25 11:03:59.515923294 (475h34m59.387006329s) [upload=417450]: Got next segment. Inline: false Dec 25 11:03:59.515992456 (475h34m59.387075491s) [upload=417450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:03:59.515997395 (475h34m59.38708043s) [upload=417450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:03:59.763382274 (475h34m59.634465309s) [seg_pos={0 0}, upload=417450]: Begin upload segment... Dec 25 11:03:59.763426679 (475h34m59.634509724s) [seg_pos={0 0}, upload=417450]: Done begin upload segment. Dec 25 11:03:59.763452508 (475h34m59.634535543s) [noise=true , address=145.79.227.37:4567 , piece_id=PV7WZX2MYWKBW6SC5R6XKJEKMECQ7E45IWI3WGCN2UJCYRMQ3BBA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.763553290 (475h34m59.634636325s) [noise=true , address=145.79.227.52:4447 , piece_id=NGMJBLCAJ5OKYWLT5Q7MG3C6NRU7WZZQVDOB77BJH6J5GX6NO6KQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764047371 (475h34m59.635130406s) [noise=true , address=45.91.116.4:4937 , piece_id=64AINFG4GLFSYFIGEZ4JXDD2EECEAFRENLILEARTFS3L4WNBDISQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764408799 (475h34m59.635491834s) [noise=true , address=145.79.227.47:4266 , piece_id=42YSOQM5OG6FPIJ4LLFLNYGCCAT4X6YNDA4IH45AZDQFLZDXW5PQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764673713 (475h34m59.635756748s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764692268 (475h34m59.635775303s) [noise=true , address=145.79.227.49:4338 , piece_id=KEDFC64SQTVWKRANH6PBRB4UIHULTTO2JLHXDZDUZHTAGYVSI7QQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764969907 (475h34m59.636052942s) [noise=true , address=77.90.31.21:4880 , piece_id=3UNC5SGWF6K5PZV24W5ZR6E7VA4XTP4Z3L64RH5ZDK6LITGDRBOQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765257254 (475h34m59.636340289s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765284636 (475h34m59.636367671s) [noise=true , address=145.79.227.6:4258 , piece_id=S6GJBYDLOGHYMH234TEX3ZZ7QYPHPKAGUWZLF3VDX3Q7DQNCEBQQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765664510 (475h34m59.636747555s) [noise=true , address=145.79.227.4:4187 , piece_id=C4A5R2WGU2D7MCLK2ZITZKAX4JA37C6V63JQLRZ6XZ3PYRVMRUUA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765678806 (475h34m59.636761841s) [noise=true , address=145.79.227.38:4977 , piece_id=NS6W2YFFCWZCTS74C2NGJTFJIYEMP55L2SBUVDB2ZDUL2EM2WT4A, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765969811 (475h34m59.637052846s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766154823 (475h34m59.637237868s) [noise=true , address=145.79.227.7:4294 , piece_id=VB2I6ORRUF5KFQZDI2NMUP75JAA4VH5EX5IAW74WYRICLAEZ57ZA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766554695 (475h34m59.63763773s) [noise=true , address=77.90.31.13:4627 , piece_id=5JFHJ4HUVYR27UTCPBXMPARMLXSMYOSTTYTI6FGEBI4KDTT7L7WA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766958664 (475h34m59.638041699s) [noise=true , address=145.79.227.42:4121 , piece_id=HUXHLCXPGXU6LGZBVT5B2GIMLHFJDYAV3HDPJKXM3RPT5OS7BGIQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.767318640 (475h34m59.638401675s) [noise=true , address=45.91.116.14:4191 , piece_id=XA53IWCSYI2VKZZPX7FWDZM5N5CXCWPCCA7F2NHML6MTBR2MSZYQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:04:01.388268928 (475h35m1.259351973s) [upload=417450]: Next returned nil segment Dec 25 11:04:01.388286351 (475h35m1.259369406s) [upload=417450]: Waiting for error group managing segments... Dec 25 11:04:01.773288615 (475h35m1.64437166s) [noise=true , address=145.79.227.37:4567 , piece_id=PV7WZX2MYWKBW6SC5R6XKJEKMECQ7E45IWI3WGCN2UJCYRMQ3BBA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.774294370 (475h35m1.645377415s) [noise=true , address=145.79.227.38:4977 , piece_id=NS6W2YFFCWZCTS74C2NGJTFJIYEMP55L2SBUVDB2ZDUL2EM2WT4A, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.777947667 (475h35m1.649030712s) [noise=true , address=145.79.227.42:4121 , piece_id=HUXHLCXPGXU6LGZBVT5B2GIMLHFJDYAV3HDPJKXM3RPT5OS7BGIQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.783365162 (475h35m1.654448207s) [noise=true , address=145.79.227.52:4447 , piece_id=NGMJBLCAJ5OKYWLT5Q7MG3C6NRU7WZZQVDOB77BJH6J5GX6NO6KQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.783464501 (475h35m1.654547546s) [noise=true , address=145.79.227.47:4266 , piece_id=42YSOQM5OG6FPIJ4LLFLNYGCCAT4X6YNDA4IH45AZDQFLZDXW5PQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.784244136 (475h35m1.655327181s) [noise=true , address=145.79.227.49:4338 , piece_id=KEDFC64SQTVWKRANH6PBRB4UIHULTTO2JLHXDZDUZHTAGYVSI7QQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.790940567 (475h35m1.662023612s) [noise=true , address=145.79.227.7:4294 , piece_id=VB2I6ORRUF5KFQZDI2NMUP75JAA4VH5EX5IAW74WYRICLAEZ57ZA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.799823582 (475h35m1.670906627s) [noise=true , address=145.79.227.6:4258 , piece_id=S6GJBYDLOGHYMH234TEX3ZZ7QYPHPKAGUWZLF3VDX3Q7DQNCEBQQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.830799615 (475h35m1.70188266s) [noise=true , address=145.79.227.4:4187 , piece_id=C4A5R2WGU2D7MCLK2ZITZKAX4JA37C6V63JQLRZ6XZ3PYRVMRUUA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.016322589 (475h35m1.887405634s) [noise=true , address=77.90.31.13:4627 , piece_id=5JFHJ4HUVYR27UTCPBXMPARMLXSMYOSTTYTI6FGEBI4KDTT7L7WA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.041203226 (475h35m1.912286261s) [noise=true , address=45.91.116.4:4937 , piece_id=64AINFG4GLFSYFIGEZ4JXDD2EECEAFRENLILEARTFS3L4WNBDISQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.052992725 (475h35m1.92407577s) [noise=true , address=45.91.116.14:4191 , piece_id=XA53IWCSYI2VKZZPX7FWDZM5N5CXCWPCCA7F2NHML6MTBR2MSZYQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.072943965 (475h35m1.94402702s) [noise=true , address=77.90.31.21:4880 , piece_id=3UNC5SGWF6K5PZV24W5ZR6E7VA4XTP4Z3L64RH5ZDK6LITGDRBOQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.072967100 (475h35m1.944050155s) [seg_pos={0 0}, upload=417450]: Segment reached optimal threshold of 13 pieces. Dec 25 11:04:02.073232575 (475h35m1.94431562s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073244077 (475h35m1.944327122s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073394233 (475h35m1.944477288s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073405784 (475h35m1.94448884s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073773144 (475h35m1.944856199s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073784846 (475h35m1.944867891s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073924402 (475h35m1.945007447s) [seg_pos={0 0}, upload=417450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:04:02.074031245 (475h35m1.945114291s) [upload=417450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:04:02.074038449 (475h35m1.945121494s) [upload=417450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:04:02.360149859 (475h35m2.231232915s) [upload=417450]: Done uploading segments... Dec 25 11:04:02.360212960 (475h35m2.231296015s) [upload=417450]: Upload finished. err: <nil> Duration: 5.776218582s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................X...........XXX...................XX..X..] duration: 6.145639908s trace: https://jaeger.dev.storj.io/trace/2dc5d425884c83bb FAIL (2025-12-25 11:05:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 11:03:59.515608956 (475h34m59.386692001s) [upload=417450]: Starting upload Dec 25 11:03:59.515687064 (475h34m59.386770119s) [upload=417450]: Done starting upload Dec 25 11:03:59.515805179 (475h34m59.386888224s) [upload=417450]: Uploading segments... Dec 25 11:03:59.515923294 (475h34m59.387006329s) [upload=417450]: Got next segment. Inline: false Dec 25 11:03:59.515992456 (475h34m59.387075491s) [upload=417450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:03:59.515997395 (475h34m59.38708043s) [upload=417450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:03:59.763382274 (475h34m59.634465309s) [seg_pos={0 0}, upload=417450]: Begin upload segment... Dec 25 11:03:59.763426679 (475h34m59.634509724s) [seg_pos={0 0}, upload=417450]: Done begin upload segment. Dec 25 11:03:59.763452508 (475h34m59.634535543s) [noise=true , address=145.79.227.37:4567 , piece_id=PV7WZX2MYWKBW6SC5R6XKJEKMECQ7E45IWI3WGCN2UJCYRMQ3BBA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.763553290 (475h34m59.634636325s) [noise=true , address=145.79.227.52:4447 , piece_id=NGMJBLCAJ5OKYWLT5Q7MG3C6NRU7WZZQVDOB77BJH6J5GX6NO6KQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764047371 (475h34m59.635130406s) [noise=true , address=45.91.116.4:4937 , piece_id=64AINFG4GLFSYFIGEZ4JXDD2EECEAFRENLILEARTFS3L4WNBDISQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764408799 (475h34m59.635491834s) [noise=true , address=145.79.227.47:4266 , piece_id=42YSOQM5OG6FPIJ4LLFLNYGCCAT4X6YNDA4IH45AZDQFLZDXW5PQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764673713 (475h34m59.635756748s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764692268 (475h34m59.635775303s) [noise=true , address=145.79.227.49:4338 , piece_id=KEDFC64SQTVWKRANH6PBRB4UIHULTTO2JLHXDZDUZHTAGYVSI7QQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764969907 (475h34m59.636052942s) [noise=true , address=77.90.31.21:4880 , piece_id=3UNC5SGWF6K5PZV24W5ZR6E7VA4XTP4Z3L64RH5ZDK6LITGDRBOQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765257254 (475h34m59.636340289s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765284636 (475h34m59.636367671s) [noise=true , address=145.79.227.6:4258 , piece_id=S6GJBYDLOGHYMH234TEX3ZZ7QYPHPKAGUWZLF3VDX3Q7DQNCEBQQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765664510 (475h34m59.636747555s) [noise=true , address=145.79.227.4:4187 , piece_id=C4A5R2WGU2D7MCLK2ZITZKAX4JA37C6V63JQLRZ6XZ3PYRVMRUUA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765678806 (475h34m59.636761841s) [noise=true , address=145.79.227.38:4977 , piece_id=NS6W2YFFCWZCTS74C2NGJTFJIYEMP55L2SBUVDB2ZDUL2EM2WT4A, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765969811 (475h34m59.637052846s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766154823 (475h34m59.637237868s) [noise=true , address=145.79.227.7:4294 , piece_id=VB2I6ORRUF5KFQZDI2NMUP75JAA4VH5EX5IAW74WYRICLAEZ57ZA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766554695 (475h34m59.63763773s) [noise=true , address=77.90.31.13:4627 , piece_id=5JFHJ4HUVYR27UTCPBXMPARMLXSMYOSTTYTI6FGEBI4KDTT7L7WA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766958664 (475h34m59.638041699s) [noise=true , address=145.79.227.42:4121 , piece_id=HUXHLCXPGXU6LGZBVT5B2GIMLHFJDYAV3HDPJKXM3RPT5OS7BGIQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.767318640 (475h34m59.638401675s) [noise=true , address=45.91.116.14:4191 , piece_id=XA53IWCSYI2VKZZPX7FWDZM5N5CXCWPCCA7F2NHML6MTBR2MSZYQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:04:01.388268928 (475h35m1.259351973s) [upload=417450]: Next returned nil segment Dec 25 11:04:01.388286351 (475h35m1.259369406s) [upload=417450]: Waiting for error group managing segments... Dec 25 11:04:01.773288615 (475h35m1.64437166s) [noise=true , address=145.79.227.37:4567 , piece_id=PV7WZX2MYWKBW6SC5R6XKJEKMECQ7E45IWI3WGCN2UJCYRMQ3BBA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.774294370 (475h35m1.645377415s) [noise=true , address=145.79.227.38:4977 , piece_id=NS6W2YFFCWZCTS74C2NGJTFJIYEMP55L2SBUVDB2ZDUL2EM2WT4A, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.777947667 (475h35m1.649030712s) [noise=true , address=145.79.227.42:4121 , piece_id=HUXHLCXPGXU6LGZBVT5B2GIMLHFJDYAV3HDPJKXM3RPT5OS7BGIQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.783365162 (475h35m1.654448207s) [noise=true , address=145.79.227.52:4447 , piece_id=NGMJBLCAJ5OKYWLT5Q7MG3C6NRU7WZZQVDOB77BJH6J5GX6NO6KQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.783464501 (475h35m1.654547546s) [noise=true , address=145.79.227.47:4266 , piece_id=42YSOQM5OG6FPIJ4LLFLNYGCCAT4X6YNDA4IH45AZDQFLZDXW5PQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.784244136 (475h35m1.655327181s) [noise=true , address=145.79.227.49:4338 , piece_id=KEDFC64SQTVWKRANH6PBRB4UIHULTTO2JLHXDZDUZHTAGYVSI7QQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.790940567 (475h35m1.662023612s) [noise=true , address=145.79.227.7:4294 , piece_id=VB2I6ORRUF5KFQZDI2NMUP75JAA4VH5EX5IAW74WYRICLAEZ57ZA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.799823582 (475h35m1.670906627s) [noise=true , address=145.79.227.6:4258 , piece_id=S6GJBYDLOGHYMH234TEX3ZZ7QYPHPKAGUWZLF3VDX3Q7DQNCEBQQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.830799615 (475h35m1.70188266s) [noise=true , address=145.79.227.4:4187 , piece_id=C4A5R2WGU2D7MCLK2ZITZKAX4JA37C6V63JQLRZ6XZ3PYRVMRUUA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.016322589 (475h35m1.887405634s) [noise=true , address=77.90.31.13:4627 , piece_id=5JFHJ4HUVYR27UTCPBXMPARMLXSMYOSTTYTI6FGEBI4KDTT7L7WA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.041203226 (475h35m1.912286261s) [noise=true , address=45.91.116.4:4937 , piece_id=64AINFG4GLFSYFIGEZ4JXDD2EECEAFRENLILEARTFS3L4WNBDISQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.052992725 (475h35m1.92407577s) [noise=true , address=45.91.116.14:4191 , piece_id=XA53IWCSYI2VKZZPX7FWDZM5N5CXCWPCCA7F2NHML6MTBR2MSZYQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.072943965 (475h35m1.94402702s) [noise=true , address=77.90.31.21:4880 , piece_id=3UNC5SGWF6K5PZV24W5ZR6E7VA4XTP4Z3L64RH5ZDK6LITGDRBOQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.072967100 (475h35m1.944050155s) [seg_pos={0 0}, upload=417450]: Segment reached optimal threshold of 13 pieces. Dec 25 11:04:02.073232575 (475h35m1.94431562s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073244077 (475h35m1.944327122s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073394233 (475h35m1.944477288s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073405784 (475h35m1.94448884s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073773144 (475h35m1.944856199s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073784846 (475h35m1.944867891s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073924402 (475h35m1.945007447s) [seg_pos={0 0}, upload=417450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:04:02.074031245 (475h35m1.945114291s) [upload=417450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:04:02.074038449 (475h35m1.945121494s) [upload=417450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:04:02.360149859 (475h35m2.231232915s) [upload=417450]: Done uploading segments... Dec 25 11:04:02.360212960 (475h35m2.231296015s) [upload=417450]: Upload finished. err: <nil> Duration: 6.145666088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................X...........XXX...................XX..X.] duration: 6.053014215s trace: https://jaeger.dev.storj.io/trace/2dc5d425884c83bb FAIL (2025-12-25 11:04:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 11:03:59.515608956 (475h34m59.386692001s) [upload=417450]: Starting upload Dec 25 11:03:59.515687064 (475h34m59.386770119s) [upload=417450]: Done starting upload Dec 25 11:03:59.515805179 (475h34m59.386888224s) [upload=417450]: Uploading segments... Dec 25 11:03:59.515923294 (475h34m59.387006329s) [upload=417450]: Got next segment. Inline: false Dec 25 11:03:59.515992456 (475h34m59.387075491s) [upload=417450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:03:59.515997395 (475h34m59.38708043s) [upload=417450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:03:59.763382274 (475h34m59.634465309s) [seg_pos={0 0}, upload=417450]: Begin upload segment... Dec 25 11:03:59.763426679 (475h34m59.634509724s) [seg_pos={0 0}, upload=417450]: Done begin upload segment. Dec 25 11:03:59.763452508 (475h34m59.634535543s) [noise=true , address=145.79.227.37:4567 , piece_id=PV7WZX2MYWKBW6SC5R6XKJEKMECQ7E45IWI3WGCN2UJCYRMQ3BBA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.763553290 (475h34m59.634636325s) [noise=true , address=145.79.227.52:4447 , piece_id=NGMJBLCAJ5OKYWLT5Q7MG3C6NRU7WZZQVDOB77BJH6J5GX6NO6KQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764047371 (475h34m59.635130406s) [noise=true , address=45.91.116.4:4937 , piece_id=64AINFG4GLFSYFIGEZ4JXDD2EECEAFRENLILEARTFS3L4WNBDISQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764408799 (475h34m59.635491834s) [noise=true , address=145.79.227.47:4266 , piece_id=42YSOQM5OG6FPIJ4LLFLNYGCCAT4X6YNDA4IH45AZDQFLZDXW5PQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764673713 (475h34m59.635756748s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764692268 (475h34m59.635775303s) [noise=true , address=145.79.227.49:4338 , piece_id=KEDFC64SQTVWKRANH6PBRB4UIHULTTO2JLHXDZDUZHTAGYVSI7QQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764969907 (475h34m59.636052942s) [noise=true , address=77.90.31.21:4880 , piece_id=3UNC5SGWF6K5PZV24W5ZR6E7VA4XTP4Z3L64RH5ZDK6LITGDRBOQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765257254 (475h34m59.636340289s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765284636 (475h34m59.636367671s) [noise=true , address=145.79.227.6:4258 , piece_id=S6GJBYDLOGHYMH234TEX3ZZ7QYPHPKAGUWZLF3VDX3Q7DQNCEBQQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765664510 (475h34m59.636747555s) [noise=true , address=145.79.227.4:4187 , piece_id=C4A5R2WGU2D7MCLK2ZITZKAX4JA37C6V63JQLRZ6XZ3PYRVMRUUA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765678806 (475h34m59.636761841s) [noise=true , address=145.79.227.38:4977 , piece_id=NS6W2YFFCWZCTS74C2NGJTFJIYEMP55L2SBUVDB2ZDUL2EM2WT4A, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765969811 (475h34m59.637052846s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766154823 (475h34m59.637237868s) [noise=true , address=145.79.227.7:4294 , piece_id=VB2I6ORRUF5KFQZDI2NMUP75JAA4VH5EX5IAW74WYRICLAEZ57ZA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766554695 (475h34m59.63763773s) [noise=true , address=77.90.31.13:4627 , piece_id=5JFHJ4HUVYR27UTCPBXMPARMLXSMYOSTTYTI6FGEBI4KDTT7L7WA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766958664 (475h34m59.638041699s) [noise=true , address=145.79.227.42:4121 , piece_id=HUXHLCXPGXU6LGZBVT5B2GIMLHFJDYAV3HDPJKXM3RPT5OS7BGIQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.767318640 (475h34m59.638401675s) [noise=true , address=45.91.116.14:4191 , piece_id=XA53IWCSYI2VKZZPX7FWDZM5N5CXCWPCCA7F2NHML6MTBR2MSZYQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:04:01.388268928 (475h35m1.259351973s) [upload=417450]: Next returned nil segment Dec 25 11:04:01.388286351 (475h35m1.259369406s) [upload=417450]: Waiting for error group managing segments... Dec 25 11:04:01.773288615 (475h35m1.64437166s) [noise=true , address=145.79.227.37:4567 , piece_id=PV7WZX2MYWKBW6SC5R6XKJEKMECQ7E45IWI3WGCN2UJCYRMQ3BBA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.774294370 (475h35m1.645377415s) [noise=true , address=145.79.227.38:4977 , piece_id=NS6W2YFFCWZCTS74C2NGJTFJIYEMP55L2SBUVDB2ZDUL2EM2WT4A, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.777947667 (475h35m1.649030712s) [noise=true , address=145.79.227.42:4121 , piece_id=HUXHLCXPGXU6LGZBVT5B2GIMLHFJDYAV3HDPJKXM3RPT5OS7BGIQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.783365162 (475h35m1.654448207s) [noise=true , address=145.79.227.52:4447 , piece_id=NGMJBLCAJ5OKYWLT5Q7MG3C6NRU7WZZQVDOB77BJH6J5GX6NO6KQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.783464501 (475h35m1.654547546s) [noise=true , address=145.79.227.47:4266 , piece_id=42YSOQM5OG6FPIJ4LLFLNYGCCAT4X6YNDA4IH45AZDQFLZDXW5PQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.784244136 (475h35m1.655327181s) [noise=true , address=145.79.227.49:4338 , piece_id=KEDFC64SQTVWKRANH6PBRB4UIHULTTO2JLHXDZDUZHTAGYVSI7QQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.790940567 (475h35m1.662023612s) [noise=true , address=145.79.227.7:4294 , piece_id=VB2I6ORRUF5KFQZDI2NMUP75JAA4VH5EX5IAW74WYRICLAEZ57ZA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.799823582 (475h35m1.670906627s) [noise=true , address=145.79.227.6:4258 , piece_id=S6GJBYDLOGHYMH234TEX3ZZ7QYPHPKAGUWZLF3VDX3Q7DQNCEBQQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.830799615 (475h35m1.70188266s) [noise=true , address=145.79.227.4:4187 , piece_id=C4A5R2WGU2D7MCLK2ZITZKAX4JA37C6V63JQLRZ6XZ3PYRVMRUUA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.016322589 (475h35m1.887405634s) [noise=true , address=77.90.31.13:4627 , piece_id=5JFHJ4HUVYR27UTCPBXMPARMLXSMYOSTTYTI6FGEBI4KDTT7L7WA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.041203226 (475h35m1.912286261s) [noise=true , address=45.91.116.4:4937 , piece_id=64AINFG4GLFSYFIGEZ4JXDD2EECEAFRENLILEARTFS3L4WNBDISQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.052992725 (475h35m1.92407577s) [noise=true , address=45.91.116.14:4191 , piece_id=XA53IWCSYI2VKZZPX7FWDZM5N5CXCWPCCA7F2NHML6MTBR2MSZYQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.072943965 (475h35m1.94402702s) [noise=true , address=77.90.31.21:4880 , piece_id=3UNC5SGWF6K5PZV24W5ZR6E7VA4XTP4Z3L64RH5ZDK6LITGDRBOQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.072967100 (475h35m1.944050155s) [seg_pos={0 0}, upload=417450]: Segment reached optimal threshold of 13 pieces. Dec 25 11:04:02.073232575 (475h35m1.94431562s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073244077 (475h35m1.944327122s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073394233 (475h35m1.944477288s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073405784 (475h35m1.94448884s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073773144 (475h35m1.944856199s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073784846 (475h35m1.944867891s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073924402 (475h35m1.945007447s) [seg_pos={0 0}, upload=417450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:04:02.074031245 (475h35m1.945114291s) [upload=417450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:04:02.074038449 (475h35m1.945121494s) [upload=417450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:04:02.360149859 (475h35m2.231232915s) [upload=417450]: Done uploading segments... Dec 25 11:04:02.360212960 (475h35m2.231296015s) [upload=417450]: Upload finished. err: <nil> Duration: 6.053040824s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................X...........XXX...................XX..X] duration: 30.003349236s trace: https://jaeger.dev.storj.io/trace/2dc5d425884c83bb FAIL (2025-12-25 11:04:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 11:03:59.515608956 (475h34m59.386692001s) [upload=417450]: Starting upload Dec 25 11:03:59.515687064 (475h34m59.386770119s) [upload=417450]: Done starting upload Dec 25 11:03:59.515805179 (475h34m59.386888224s) [upload=417450]: Uploading segments... Dec 25 11:03:59.515923294 (475h34m59.387006329s) [upload=417450]: Got next segment. Inline: false Dec 25 11:03:59.515992456 (475h34m59.387075491s) [upload=417450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:03:59.515997395 (475h34m59.38708043s) [upload=417450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:03:59.763382274 (475h34m59.634465309s) [seg_pos={0 0}, upload=417450]: Begin upload segment... Dec 25 11:03:59.763426679 (475h34m59.634509724s) [seg_pos={0 0}, upload=417450]: Done begin upload segment. Dec 25 11:03:59.763452508 (475h34m59.634535543s) [noise=true , address=145.79.227.37:4567 , piece_id=PV7WZX2MYWKBW6SC5R6XKJEKMECQ7E45IWI3WGCN2UJCYRMQ3BBA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.763553290 (475h34m59.634636325s) [noise=true , address=145.79.227.52:4447 , piece_id=NGMJBLCAJ5OKYWLT5Q7MG3C6NRU7WZZQVDOB77BJH6J5GX6NO6KQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764047371 (475h34m59.635130406s) [noise=true , address=45.91.116.4:4937 , piece_id=64AINFG4GLFSYFIGEZ4JXDD2EECEAFRENLILEARTFS3L4WNBDISQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764408799 (475h34m59.635491834s) [noise=true , address=145.79.227.47:4266 , piece_id=42YSOQM5OG6FPIJ4LLFLNYGCCAT4X6YNDA4IH45AZDQFLZDXW5PQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764673713 (475h34m59.635756748s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764692268 (475h34m59.635775303s) [noise=true , address=145.79.227.49:4338 , piece_id=KEDFC64SQTVWKRANH6PBRB4UIHULTTO2JLHXDZDUZHTAGYVSI7QQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.764969907 (475h34m59.636052942s) [noise=true , address=77.90.31.21:4880 , piece_id=3UNC5SGWF6K5PZV24W5ZR6E7VA4XTP4Z3L64RH5ZDK6LITGDRBOQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765257254 (475h34m59.636340289s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765284636 (475h34m59.636367671s) [noise=true , address=145.79.227.6:4258 , piece_id=S6GJBYDLOGHYMH234TEX3ZZ7QYPHPKAGUWZLF3VDX3Q7DQNCEBQQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765664510 (475h34m59.636747555s) [noise=true , address=145.79.227.4:4187 , piece_id=C4A5R2WGU2D7MCLK2ZITZKAX4JA37C6V63JQLRZ6XZ3PYRVMRUUA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765678806 (475h34m59.636761841s) [noise=true , address=145.79.227.38:4977 , piece_id=NS6W2YFFCWZCTS74C2NGJTFJIYEMP55L2SBUVDB2ZDUL2EM2WT4A, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.765969811 (475h34m59.637052846s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766154823 (475h34m59.637237868s) [noise=true , address=145.79.227.7:4294 , piece_id=VB2I6ORRUF5KFQZDI2NMUP75JAA4VH5EX5IAW74WYRICLAEZ57ZA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766554695 (475h34m59.63763773s) [noise=true , address=77.90.31.13:4627 , piece_id=5JFHJ4HUVYR27UTCPBXMPARMLXSMYOSTTYTI6FGEBI4KDTT7L7WA, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.766958664 (475h34m59.638041699s) [noise=true , address=145.79.227.42:4121 , piece_id=HUXHLCXPGXU6LGZBVT5B2GIMLHFJDYAV3HDPJKXM3RPT5OS7BGIQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:03:59.767318640 (475h34m59.638401675s) [noise=true , address=45.91.116.14:4191 , piece_id=XA53IWCSYI2VKZZPX7FWDZM5N5CXCWPCCA7F2NHML6MTBR2MSZYQ, seg_pos={0 0}, upload=417450]: Uploading piece... Dec 25 11:04:01.388268928 (475h35m1.259351973s) [upload=417450]: Next returned nil segment Dec 25 11:04:01.388286351 (475h35m1.259369406s) [upload=417450]: Waiting for error group managing segments... Dec 25 11:04:01.773288615 (475h35m1.64437166s) [noise=true , address=145.79.227.37:4567 , piece_id=PV7WZX2MYWKBW6SC5R6XKJEKMECQ7E45IWI3WGCN2UJCYRMQ3BBA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.774294370 (475h35m1.645377415s) [noise=true , address=145.79.227.38:4977 , piece_id=NS6W2YFFCWZCTS74C2NGJTFJIYEMP55L2SBUVDB2ZDUL2EM2WT4A, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.777947667 (475h35m1.649030712s) [noise=true , address=145.79.227.42:4121 , piece_id=HUXHLCXPGXU6LGZBVT5B2GIMLHFJDYAV3HDPJKXM3RPT5OS7BGIQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.783365162 (475h35m1.654448207s) [noise=true , address=145.79.227.52:4447 , piece_id=NGMJBLCAJ5OKYWLT5Q7MG3C6NRU7WZZQVDOB77BJH6J5GX6NO6KQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.783464501 (475h35m1.654547546s) [noise=true , address=145.79.227.47:4266 , piece_id=42YSOQM5OG6FPIJ4LLFLNYGCCAT4X6YNDA4IH45AZDQFLZDXW5PQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.784244136 (475h35m1.655327181s) [noise=true , address=145.79.227.49:4338 , piece_id=KEDFC64SQTVWKRANH6PBRB4UIHULTTO2JLHXDZDUZHTAGYVSI7QQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.790940567 (475h35m1.662023612s) [noise=true , address=145.79.227.7:4294 , piece_id=VB2I6ORRUF5KFQZDI2NMUP75JAA4VH5EX5IAW74WYRICLAEZ57ZA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.799823582 (475h35m1.670906627s) [noise=true , address=145.79.227.6:4258 , piece_id=S6GJBYDLOGHYMH234TEX3ZZ7QYPHPKAGUWZLF3VDX3Q7DQNCEBQQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:01.830799615 (475h35m1.70188266s) [noise=true , address=145.79.227.4:4187 , piece_id=C4A5R2WGU2D7MCLK2ZITZKAX4JA37C6V63JQLRZ6XZ3PYRVMRUUA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.016322589 (475h35m1.887405634s) [noise=true , address=77.90.31.13:4627 , piece_id=5JFHJ4HUVYR27UTCPBXMPARMLXSMYOSTTYTI6FGEBI4KDTT7L7WA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.041203226 (475h35m1.912286261s) [noise=true , address=45.91.116.4:4937 , piece_id=64AINFG4GLFSYFIGEZ4JXDD2EECEAFRENLILEARTFS3L4WNBDISQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.052992725 (475h35m1.92407577s) [noise=true , address=45.91.116.14:4191 , piece_id=XA53IWCSYI2VKZZPX7FWDZM5N5CXCWPCCA7F2NHML6MTBR2MSZYQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.072943965 (475h35m1.94402702s) [noise=true , address=77.90.31.21:4880 , piece_id=3UNC5SGWF6K5PZV24W5ZR6E7VA4XTP4Z3L64RH5ZDK6LITGDRBOQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: <nil> Dec 25 11:04:02.072967100 (475h35m1.944050155s) [seg_pos={0 0}, upload=417450]: Segment reached optimal threshold of 13 pieces. Dec 25 11:04:02.073232575 (475h35m1.94431562s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073244077 (475h35m1.944327122s) [noise=true , address=77.90.31.16:4735 , piece_id=GEYVM6PASJAHF2EW4NS3J3TA7CRXODZ2BBKSVCXLTHTEDGDFFFSQ, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073394233 (475h35m1.944477288s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073405784 (475h35m1.94448884s) [noise=true , address=45.91.116.11:4119 , piece_id=766BZN7EPIP66IQTUO7DS6IMPLJN3ISY6IC52QAYTZD2KI3I5ZGQ, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073773144 (475h35m1.944856199s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 25 11:04:02.073784846 (475h35m1.944867891s) [noise=true , address=145.79.227.33:4871 , piece_id=L7WSYCT7XT7RUSX2C7F36TCV62RADTAQKSMWXUTADGI3E45MRMRA, seg_pos={0 0}, upload=417450]: Long tail cancellation due to optimal threshold Dec 25 11:04:02.073924402 (475h35m1.945007447s) [seg_pos={0 0}, upload=417450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:04:02.074031245 (475h35m1.945114291s) [upload=417450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:04:02.074038449 (475h35m1.945121494s) [upload=417450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:04:02.360149859 (475h35m2.231232915s) [upload=417450]: Done uploading segments... Dec 25 11:04:02.360212960 (475h35m2.231296015s) [upload=417450]: Upload finished. err: <nil> Duration: 30.003388941s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................X...........XXX...................XX..] duration: 6.016536183s trace: https://jaeger.dev.storj.io/trace/63aa3abcb7162390 FAIL (2025-12-25 11:03:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 3BZ7ECWEVLGWB2Y2LCJKDOZ3CXFDCIZXFOYFFEA7HH5545OUW23A) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: HHQ2UCY7ZEXJY3PYADPOE3OB5437XTRCHBHQVPIC5HVTQURWH3PQ) context deadline exceeded ======LOGS====== Dec 25 11:01:48.147528959 (475h32m48.018612005s) [upload=417414]: Starting upload Dec 25 11:01:48.147580577 (475h32m48.018663612s) [upload=417414]: Done starting upload Dec 25 11:01:48.147784927 (475h32m48.018867962s) [upload=417414]: Uploading segments... Dec 25 11:01:48.147824953 (475h32m48.018907988s) [upload=417414]: Got next segment. Inline: false Dec 25 11:01:48.147850592 (475h32m48.018933627s) [upload=417414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:01:48.147856252 (475h32m48.018939287s) [upload=417414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:01:48.397107596 (475h32m48.268190641s) [seg_pos={0 0}, upload=417414]: Begin upload segment... Dec 25 11:01:48.397173471 (475h32m48.268256506s) [seg_pos={0 0}, upload=417414]: Done begin upload segment. Dec 25 11:01:48.397199832 (475h32m48.268282887s) [noise=true , address=145.79.227.7:4294 , piece_id=UYWASIWCIDNZZPY3BG6GCSXAXDBHMCQRJC23KPHF4CIJ2L5WVPHA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.397214750 (475h32m48.268297785s) [noise=true , address=77.90.31.18:4808 , piece_id=HHQ2UCY7ZEXJY3PYADPOE3OB5437XTRCHBHQVPIC5HVTQURWH3PQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.397274604 (475h32m48.268357649s) [noise=true , address=45.91.116.13:4156 , piece_id=XUNA3KXLLUURB5CTQYUNBGFM3E2EVAPPJXODHGXSA4TUOGSYOIRQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.397388481 (475h32m48.268471516s) [noise=true , address=45.91.116.11:4120 , piece_id=2LDWGLEITCRFVF36T3NQQEWOSPXO7EI5VTDBYIMAZCGENVY44TJA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.398181851 (475h32m48.269264897s) [noise=true , address=145.79.227.5:4222 , piece_id=4LE6IHY6PEYFLBXFWRFLYT4PW2HEZ7EC36WLRWKSZD6JJTYBMUWA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.399390383 (475h32m48.270473418s) [noise=true , address=145.79.227.39:4015 , piece_id=3Y2M3OQF56UOACUSA5NCXTIC3L3XMWTVXR7OEE5DT3MQTHO2NZFA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.399644847 (475h32m48.270727892s) [noise=true , address=145.79.227.29:4942 , piece_id=LEL7BXIDHLJA7LRFFLDMSLMY3DY7FCBZNRFFN5J7YYVV75GUYEUA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.404362159 (475h32m48.275445204s) [noise=true , address=145.79.227.19:4077 , piece_id=X4BOJFSCLCWJLZEGZ4E6X2L5SRTKVV7JD77BFOM3WV2T35IMS34A, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.406218374 (475h32m48.277301419s) [noise=true , address=45.91.116.8:4048 , piece_id=4CXNEPTHDTX6DUJIDK5G724FL5SVMCU3J6NMOEUSPKSZEMG5DKQA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.407950001 (475h32m48.279033046s) [noise=true , address=145.79.227.14:4838 , piece_id=OPOLO4TXG7KA454IEPQLTCJWYROUEFBPQ3N7JWWIP4FYYETWQ4SQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.409715563 (475h32m48.280798598s) [noise=true , address=77.90.31.14:4661 , piece_id=CO6CEUAJ7GG5AD2QKR2UYPMCBQ2KZY5FHYKF3E3HQ55PGOXQY2CQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.414699804 (475h32m48.285782849s) [noise=true , address=145.79.227.42:4121 , piece_id=XA2ETUFMFF3LTJA2VLG4YPQ7ELAVLEPDW4XRWAFAYDM6K53PA5CA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.415863701 (475h32m48.286946746s) [noise=true , address=77.90.31.21:4878 , piece_id=5XE2A4QEQT4HH3MWH4KAND357N2Z5KW2TU4GPR4UHTOUW5QZGX7A, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.419403230 (475h32m48.290486265s) [noise=true , address=77.90.31.19:4844 , piece_id=3BZ7ECWEVLGWB2Y2LCJKDOZ3CXFDCIZXFOYFFEA7HH5545OUW23A, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.419938620 (475h32m48.291021655s) [noise=true , address=145.79.227.13:4802 , piece_id=HH7C6UOJMW2JMJPMYTNIQC6XXOBS3LPJ6VV6VESQO7RSJOHIHV2A, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.422305066 (475h32m48.293388101s) [noise=true , address=145.79.227.4:4186 , piece_id=D3JFKGJQ3RELYEUZKVKLLBSYIB5NNVSW6W2P3ZGB32767JQD3IAQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:50.128120842 (475h32m49.999203888s) [upload=417414]: Next returned nil segment Dec 25 11:01:50.128149336 (475h32m49.999232371s) [upload=417414]: Waiting for error group managing segments... Dec 25 11:01:50.411965075 (475h32m50.28304811s) [noise=true , address=145.79.227.13:4802 , piece_id=HH7C6UOJMW2JMJPMYTNIQC6XXOBS3LPJ6VV6VESQO7RSJOHIHV2A, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.417192088 (475h32m50.288275123s) [noise=true , address=145.79.227.29:4942 , piece_id=LEL7BXIDHLJA7LRFFLDMSLMY3DY7FCBZNRFFN5J7YYVV75GUYEUA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.418777657 (475h32m50.289860692s) [noise=true , address=145.79.227.39:4015 , piece_id=3Y2M3OQF56UOACUSA5NCXTIC3L3XMWTVXR7OEE5DT3MQTHO2NZFA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.419529038 (475h32m50.290612073s) [noise=true , address=145.79.227.42:4121 , piece_id=XA2ETUFMFF3LTJA2VLG4YPQ7ELAVLEPDW4XRWAFAYDM6K53PA5CA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.422284014 (475h32m50.293367049s) [noise=true , address=145.79.227.7:4294 , piece_id=UYWASIWCIDNZZPY3BG6GCSXAXDBHMCQRJC23KPHF4CIJ2L5WVPHA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.423871336 (475h32m50.294954372s) [noise=true , address=145.79.227.14:4838 , piece_id=OPOLO4TXG7KA454IEPQLTCJWYROUEFBPQ3N7JWWIP4FYYETWQ4SQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.424849218 (475h32m50.295932254s) [noise=true , address=145.79.227.19:4077 , piece_id=X4BOJFSCLCWJLZEGZ4E6X2L5SRTKVV7JD77BFOM3WV2T35IMS34A, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.430240354 (475h32m50.301323399s) [noise=true , address=145.79.227.4:4186 , piece_id=D3JFKGJQ3RELYEUZKVKLLBSYIB5NNVSW6W2P3ZGB32767JQD3IAQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.430681303 (475h32m50.301764348s) [noise=true , address=145.79.227.5:4222 , piece_id=4LE6IHY6PEYFLBXFWRFLYT4PW2HEZ7EC36WLRWKSZD6JJTYBMUWA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.651681582 (475h32m50.522764627s) [noise=true , address=77.90.31.18:4808 , piece_id=HHQ2UCY7ZEXJY3PYADPOE3OB5437XTRCHBHQVPIC5HVTQURWH3PQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.670832920 (475h32m50.541915955s) [noise=true , address=77.90.31.19:4844 , piece_id=3BZ7ECWEVLGWB2Y2LCJKDOZ3CXFDCIZXFOYFFEA7HH5545OUW23A, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.689160839 (475h32m50.560243914s) [noise=true , address=45.91.116.11:4120 , piece_id=2LDWGLEITCRFVF36T3NQQEWOSPXO7EI5VTDBYIMAZCGENVY44TJA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.703849528 (475h32m50.574932574s) [noise=true , address=77.90.31.14:4661 , piece_id=CO6CEUAJ7GG5AD2QKR2UYPMCBQ2KZY5FHYKF3E3HQ55PGOXQY2CQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.703873904 (475h32m50.574956939s) [seg_pos={0 0}, upload=417414]: Segment reached optimal threshold of 13 pieces. Dec 25 11:01:50.704062734 (475h32m50.575145769s) [noise=true , address=45.91.116.13:4156 , piece_id=XUNA3KXLLUURB5CTQYUNBGFM3E2EVAPPJXODHGXSA4TUOGSYOIRQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 11:01:50.704070509 (475h32m50.575153554s) [noise=true , address=45.91.116.13:4156 , piece_id=XUNA3KXLLUURB5CTQYUNBGFM3E2EVAPPJXODHGXSA4TUOGSYOIRQ, seg_pos={0 0}, upload=417414]: Long tail cancellation due to optimal threshold Dec 25 11:01:50.704283895 (475h32m50.57536694s) [noise=true , address=77.90.31.21:4878 , piece_id=5XE2A4QEQT4HH3MWH4KAND357N2Z5KW2TU4GPR4UHTOUW5QZGX7A, seg_pos={0 0}, upload=417414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 25 11:01:50.704290407 (475h32m50.575373442s) [noise=true , address=77.90.31.21:4878 , piece_id=5XE2A4QEQT4HH3MWH4KAND357N2Z5KW2TU4GPR4UHTOUW5QZGX7A, seg_pos={0 0}, upload=417414]: Long tail cancellation due to optimal threshold Dec 25 11:01:50.704385759 (475h32m50.575468804s) [noise=true , address=45.91.116.8:4048 , piece_id=4CXNEPTHDTX6DUJIDK5G724FL5SVMCU3J6NMOEUSPKSZEMG5DKQA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 25 11:01:50.704408743 (475h32m50.575491788s) [noise=true , address=45.91.116.8:4048 , piece_id=4CXNEPTHDTX6DUJIDK5G724FL5SVMCU3J6NMOEUSPKSZEMG5DKQA, seg_pos={0 0}, upload=417414]: Long tail cancellation due to optimal threshold Dec 25 11:01:50.704439111 (475h32m50.575522146s) [seg_pos={0 0}, upload=417414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:01:50.704550974 (475h32m50.575634009s) [upload=417414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:01:50.704557035 (475h32m50.57564007s) [upload=417414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:01:51.002359738 (475h32m50.873442803s) [upload=417414]: Done uploading segments... Dec 25 11:01:51.002384385 (475h32m50.87346743s) [upload=417414]: Upload finished. err: <nil> Duration: 6.016564596s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................X...........XXX...................XX.] duration: 5.755331711s trace: https://jaeger.dev.storj.io/trace/63aa3abcb7162390 FAIL (2025-12-25 11:02:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 3BZ7ECWEVLGWB2Y2LCJKDOZ3CXFDCIZXFOYFFEA7HH5545OUW23A) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: HHQ2UCY7ZEXJY3PYADPOE3OB5437XTRCHBHQVPIC5HVTQURWH3PQ) context deadline exceeded ======LOGS====== Dec 25 11:01:48.147528959 (475h32m48.018612005s) [upload=417414]: Starting upload Dec 25 11:01:48.147580577 (475h32m48.018663612s) [upload=417414]: Done starting upload Dec 25 11:01:48.147784927 (475h32m48.018867962s) [upload=417414]: Uploading segments... Dec 25 11:01:48.147824953 (475h32m48.018907988s) [upload=417414]: Got next segment. Inline: false Dec 25 11:01:48.147850592 (475h32m48.018933627s) [upload=417414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:01:48.147856252 (475h32m48.018939287s) [upload=417414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:01:48.397107596 (475h32m48.268190641s) [seg_pos={0 0}, upload=417414]: Begin upload segment... Dec 25 11:01:48.397173471 (475h32m48.268256506s) [seg_pos={0 0}, upload=417414]: Done begin upload segment. Dec 25 11:01:48.397199832 (475h32m48.268282887s) [noise=true , address=145.79.227.7:4294 , piece_id=UYWASIWCIDNZZPY3BG6GCSXAXDBHMCQRJC23KPHF4CIJ2L5WVPHA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.397214750 (475h32m48.268297785s) [noise=true , address=77.90.31.18:4808 , piece_id=HHQ2UCY7ZEXJY3PYADPOE3OB5437XTRCHBHQVPIC5HVTQURWH3PQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.397274604 (475h32m48.268357649s) [noise=true , address=45.91.116.13:4156 , piece_id=XUNA3KXLLUURB5CTQYUNBGFM3E2EVAPPJXODHGXSA4TUOGSYOIRQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.397388481 (475h32m48.268471516s) [noise=true , address=45.91.116.11:4120 , piece_id=2LDWGLEITCRFVF36T3NQQEWOSPXO7EI5VTDBYIMAZCGENVY44TJA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.398181851 (475h32m48.269264897s) [noise=true , address=145.79.227.5:4222 , piece_id=4LE6IHY6PEYFLBXFWRFLYT4PW2HEZ7EC36WLRWKSZD6JJTYBMUWA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.399390383 (475h32m48.270473418s) [noise=true , address=145.79.227.39:4015 , piece_id=3Y2M3OQF56UOACUSA5NCXTIC3L3XMWTVXR7OEE5DT3MQTHO2NZFA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.399644847 (475h32m48.270727892s) [noise=true , address=145.79.227.29:4942 , piece_id=LEL7BXIDHLJA7LRFFLDMSLMY3DY7FCBZNRFFN5J7YYVV75GUYEUA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.404362159 (475h32m48.275445204s) [noise=true , address=145.79.227.19:4077 , piece_id=X4BOJFSCLCWJLZEGZ4E6X2L5SRTKVV7JD77BFOM3WV2T35IMS34A, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.406218374 (475h32m48.277301419s) [noise=true , address=45.91.116.8:4048 , piece_id=4CXNEPTHDTX6DUJIDK5G724FL5SVMCU3J6NMOEUSPKSZEMG5DKQA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.407950001 (475h32m48.279033046s) [noise=true , address=145.79.227.14:4838 , piece_id=OPOLO4TXG7KA454IEPQLTCJWYROUEFBPQ3N7JWWIP4FYYETWQ4SQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.409715563 (475h32m48.280798598s) [noise=true , address=77.90.31.14:4661 , piece_id=CO6CEUAJ7GG5AD2QKR2UYPMCBQ2KZY5FHYKF3E3HQ55PGOXQY2CQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.414699804 (475h32m48.285782849s) [noise=true , address=145.79.227.42:4121 , piece_id=XA2ETUFMFF3LTJA2VLG4YPQ7ELAVLEPDW4XRWAFAYDM6K53PA5CA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.415863701 (475h32m48.286946746s) [noise=true , address=77.90.31.21:4878 , piece_id=5XE2A4QEQT4HH3MWH4KAND357N2Z5KW2TU4GPR4UHTOUW5QZGX7A, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.419403230 (475h32m48.290486265s) [noise=true , address=77.90.31.19:4844 , piece_id=3BZ7ECWEVLGWB2Y2LCJKDOZ3CXFDCIZXFOYFFEA7HH5545OUW23A, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.419938620 (475h32m48.291021655s) [noise=true , address=145.79.227.13:4802 , piece_id=HH7C6UOJMW2JMJPMYTNIQC6XXOBS3LPJ6VV6VESQO7RSJOHIHV2A, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.422305066 (475h32m48.293388101s) [noise=true , address=145.79.227.4:4186 , piece_id=D3JFKGJQ3RELYEUZKVKLLBSYIB5NNVSW6W2P3ZGB32767JQD3IAQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:50.128120842 (475h32m49.999203888s) [upload=417414]: Next returned nil segment Dec 25 11:01:50.128149336 (475h32m49.999232371s) [upload=417414]: Waiting for error group managing segments... Dec 25 11:01:50.411965075 (475h32m50.28304811s) [noise=true , address=145.79.227.13:4802 , piece_id=HH7C6UOJMW2JMJPMYTNIQC6XXOBS3LPJ6VV6VESQO7RSJOHIHV2A, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.417192088 (475h32m50.288275123s) [noise=true , address=145.79.227.29:4942 , piece_id=LEL7BXIDHLJA7LRFFLDMSLMY3DY7FCBZNRFFN5J7YYVV75GUYEUA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.418777657 (475h32m50.289860692s) [noise=true , address=145.79.227.39:4015 , piece_id=3Y2M3OQF56UOACUSA5NCXTIC3L3XMWTVXR7OEE5DT3MQTHO2NZFA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.419529038 (475h32m50.290612073s) [noise=true , address=145.79.227.42:4121 , piece_id=XA2ETUFMFF3LTJA2VLG4YPQ7ELAVLEPDW4XRWAFAYDM6K53PA5CA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.422284014 (475h32m50.293367049s) [noise=true , address=145.79.227.7:4294 , piece_id=UYWASIWCIDNZZPY3BG6GCSXAXDBHMCQRJC23KPHF4CIJ2L5WVPHA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.423871336 (475h32m50.294954372s) [noise=true , address=145.79.227.14:4838 , piece_id=OPOLO4TXG7KA454IEPQLTCJWYROUEFBPQ3N7JWWIP4FYYETWQ4SQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.424849218 (475h32m50.295932254s) [noise=true , address=145.79.227.19:4077 , piece_id=X4BOJFSCLCWJLZEGZ4E6X2L5SRTKVV7JD77BFOM3WV2T35IMS34A, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.430240354 (475h32m50.301323399s) [noise=true , address=145.79.227.4:4186 , piece_id=D3JFKGJQ3RELYEUZKVKLLBSYIB5NNVSW6W2P3ZGB32767JQD3IAQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.430681303 (475h32m50.301764348s) [noise=true , address=145.79.227.5:4222 , piece_id=4LE6IHY6PEYFLBXFWRFLYT4PW2HEZ7EC36WLRWKSZD6JJTYBMUWA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.651681582 (475h32m50.522764627s) [noise=true , address=77.90.31.18:4808 , piece_id=HHQ2UCY7ZEXJY3PYADPOE3OB5437XTRCHBHQVPIC5HVTQURWH3PQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.670832920 (475h32m50.541915955s) [noise=true , address=77.90.31.19:4844 , piece_id=3BZ7ECWEVLGWB2Y2LCJKDOZ3CXFDCIZXFOYFFEA7HH5545OUW23A, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.689160839 (475h32m50.560243914s) [noise=true , address=45.91.116.11:4120 , piece_id=2LDWGLEITCRFVF36T3NQQEWOSPXO7EI5VTDBYIMAZCGENVY44TJA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.703849528 (475h32m50.574932574s) [noise=true , address=77.90.31.14:4661 , piece_id=CO6CEUAJ7GG5AD2QKR2UYPMCBQ2KZY5FHYKF3E3HQ55PGOXQY2CQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.703873904 (475h32m50.574956939s) [seg_pos={0 0}, upload=417414]: Segment reached optimal threshold of 13 pieces. Dec 25 11:01:50.704062734 (475h32m50.575145769s) [noise=true , address=45.91.116.13:4156 , piece_id=XUNA3KXLLUURB5CTQYUNBGFM3E2EVAPPJXODHGXSA4TUOGSYOIRQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 11:01:50.704070509 (475h32m50.575153554s) [noise=true , address=45.91.116.13:4156 , piece_id=XUNA3KXLLUURB5CTQYUNBGFM3E2EVAPPJXODHGXSA4TUOGSYOIRQ, seg_pos={0 0}, upload=417414]: Long tail cancellation due to optimal threshold Dec 25 11:01:50.704283895 (475h32m50.57536694s) [noise=true , address=77.90.31.21:4878 , piece_id=5XE2A4QEQT4HH3MWH4KAND357N2Z5KW2TU4GPR4UHTOUW5QZGX7A, seg_pos={0 0}, upload=417414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 25 11:01:50.704290407 (475h32m50.575373442s) [noise=true , address=77.90.31.21:4878 , piece_id=5XE2A4QEQT4HH3MWH4KAND357N2Z5KW2TU4GPR4UHTOUW5QZGX7A, seg_pos={0 0}, upload=417414]: Long tail cancellation due to optimal threshold Dec 25 11:01:50.704385759 (475h32m50.575468804s) [noise=true , address=45.91.116.8:4048 , piece_id=4CXNEPTHDTX6DUJIDK5G724FL5SVMCU3J6NMOEUSPKSZEMG5DKQA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 25 11:01:50.704408743 (475h32m50.575491788s) [noise=true , address=45.91.116.8:4048 , piece_id=4CXNEPTHDTX6DUJIDK5G724FL5SVMCU3J6NMOEUSPKSZEMG5DKQA, seg_pos={0 0}, upload=417414]: Long tail cancellation due to optimal threshold Dec 25 11:01:50.704439111 (475h32m50.575522146s) [seg_pos={0 0}, upload=417414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:01:50.704550974 (475h32m50.575634009s) [upload=417414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:01:50.704557035 (475h32m50.57564007s) [upload=417414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:01:51.002359738 (475h32m50.873442803s) [upload=417414]: Done uploading segments... Dec 25 11:01:51.002384385 (475h32m50.87346743s) [upload=417414]: Upload finished. err: <nil> Duration: 5.755355396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.......................X...........XXX...................XX] duration: 30.005072245s trace: https://jaeger.dev.storj.io/trace/63aa3abcb7162390 FAIL (2025-12-25 11:02:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 3BZ7ECWEVLGWB2Y2LCJKDOZ3CXFDCIZXFOYFFEA7HH5545OUW23A) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: HHQ2UCY7ZEXJY3PYADPOE3OB5437XTRCHBHQVPIC5HVTQURWH3PQ) context deadline exceeded ======LOGS====== Dec 25 11:01:48.147528959 (475h32m48.018612005s) [upload=417414]: Starting upload Dec 25 11:01:48.147580577 (475h32m48.018663612s) [upload=417414]: Done starting upload Dec 25 11:01:48.147784927 (475h32m48.018867962s) [upload=417414]: Uploading segments... Dec 25 11:01:48.147824953 (475h32m48.018907988s) [upload=417414]: Got next segment. Inline: false Dec 25 11:01:48.147850592 (475h32m48.018933627s) [upload=417414]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:01:48.147856252 (475h32m48.018939287s) [upload=417414]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:01:48.397107596 (475h32m48.268190641s) [seg_pos={0 0}, upload=417414]: Begin upload segment... Dec 25 11:01:48.397173471 (475h32m48.268256506s) [seg_pos={0 0}, upload=417414]: Done begin upload segment. Dec 25 11:01:48.397199832 (475h32m48.268282887s) [noise=true , address=145.79.227.7:4294 , piece_id=UYWASIWCIDNZZPY3BG6GCSXAXDBHMCQRJC23KPHF4CIJ2L5WVPHA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.397214750 (475h32m48.268297785s) [noise=true , address=77.90.31.18:4808 , piece_id=HHQ2UCY7ZEXJY3PYADPOE3OB5437XTRCHBHQVPIC5HVTQURWH3PQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.397274604 (475h32m48.268357649s) [noise=true , address=45.91.116.13:4156 , piece_id=XUNA3KXLLUURB5CTQYUNBGFM3E2EVAPPJXODHGXSA4TUOGSYOIRQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.397388481 (475h32m48.268471516s) [noise=true , address=45.91.116.11:4120 , piece_id=2LDWGLEITCRFVF36T3NQQEWOSPXO7EI5VTDBYIMAZCGENVY44TJA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.398181851 (475h32m48.269264897s) [noise=true , address=145.79.227.5:4222 , piece_id=4LE6IHY6PEYFLBXFWRFLYT4PW2HEZ7EC36WLRWKSZD6JJTYBMUWA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.399390383 (475h32m48.270473418s) [noise=true , address=145.79.227.39:4015 , piece_id=3Y2M3OQF56UOACUSA5NCXTIC3L3XMWTVXR7OEE5DT3MQTHO2NZFA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.399644847 (475h32m48.270727892s) [noise=true , address=145.79.227.29:4942 , piece_id=LEL7BXIDHLJA7LRFFLDMSLMY3DY7FCBZNRFFN5J7YYVV75GUYEUA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.404362159 (475h32m48.275445204s) [noise=true , address=145.79.227.19:4077 , piece_id=X4BOJFSCLCWJLZEGZ4E6X2L5SRTKVV7JD77BFOM3WV2T35IMS34A, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.406218374 (475h32m48.277301419s) [noise=true , address=45.91.116.8:4048 , piece_id=4CXNEPTHDTX6DUJIDK5G724FL5SVMCU3J6NMOEUSPKSZEMG5DKQA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.407950001 (475h32m48.279033046s) [noise=true , address=145.79.227.14:4838 , piece_id=OPOLO4TXG7KA454IEPQLTCJWYROUEFBPQ3N7JWWIP4FYYETWQ4SQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.409715563 (475h32m48.280798598s) [noise=true , address=77.90.31.14:4661 , piece_id=CO6CEUAJ7GG5AD2QKR2UYPMCBQ2KZY5FHYKF3E3HQ55PGOXQY2CQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.414699804 (475h32m48.285782849s) [noise=true , address=145.79.227.42:4121 , piece_id=XA2ETUFMFF3LTJA2VLG4YPQ7ELAVLEPDW4XRWAFAYDM6K53PA5CA, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.415863701 (475h32m48.286946746s) [noise=true , address=77.90.31.21:4878 , piece_id=5XE2A4QEQT4HH3MWH4KAND357N2Z5KW2TU4GPR4UHTOUW5QZGX7A, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.419403230 (475h32m48.290486265s) [noise=true , address=77.90.31.19:4844 , piece_id=3BZ7ECWEVLGWB2Y2LCJKDOZ3CXFDCIZXFOYFFEA7HH5545OUW23A, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.419938620 (475h32m48.291021655s) [noise=true , address=145.79.227.13:4802 , piece_id=HH7C6UOJMW2JMJPMYTNIQC6XXOBS3LPJ6VV6VESQO7RSJOHIHV2A, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:48.422305066 (475h32m48.293388101s) [noise=true , address=145.79.227.4:4186 , piece_id=D3JFKGJQ3RELYEUZKVKLLBSYIB5NNVSW6W2P3ZGB32767JQD3IAQ, seg_pos={0 0}, upload=417414]: Uploading piece... Dec 25 11:01:50.128120842 (475h32m49.999203888s) [upload=417414]: Next returned nil segment Dec 25 11:01:50.128149336 (475h32m49.999232371s) [upload=417414]: Waiting for error group managing segments... Dec 25 11:01:50.411965075 (475h32m50.28304811s) [noise=true , address=145.79.227.13:4802 , piece_id=HH7C6UOJMW2JMJPMYTNIQC6XXOBS3LPJ6VV6VESQO7RSJOHIHV2A, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.417192088 (475h32m50.288275123s) [noise=true , address=145.79.227.29:4942 , piece_id=LEL7BXIDHLJA7LRFFLDMSLMY3DY7FCBZNRFFN5J7YYVV75GUYEUA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.418777657 (475h32m50.289860692s) [noise=true , address=145.79.227.39:4015 , piece_id=3Y2M3OQF56UOACUSA5NCXTIC3L3XMWTVXR7OEE5DT3MQTHO2NZFA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.419529038 (475h32m50.290612073s) [noise=true , address=145.79.227.42:4121 , piece_id=XA2ETUFMFF3LTJA2VLG4YPQ7ELAVLEPDW4XRWAFAYDM6K53PA5CA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.422284014 (475h32m50.293367049s) [noise=true , address=145.79.227.7:4294 , piece_id=UYWASIWCIDNZZPY3BG6GCSXAXDBHMCQRJC23KPHF4CIJ2L5WVPHA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.423871336 (475h32m50.294954372s) [noise=true , address=145.79.227.14:4838 , piece_id=OPOLO4TXG7KA454IEPQLTCJWYROUEFBPQ3N7JWWIP4FYYETWQ4SQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.424849218 (475h32m50.295932254s) [noise=true , address=145.79.227.19:4077 , piece_id=X4BOJFSCLCWJLZEGZ4E6X2L5SRTKVV7JD77BFOM3WV2T35IMS34A, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.430240354 (475h32m50.301323399s) [noise=true , address=145.79.227.4:4186 , piece_id=D3JFKGJQ3RELYEUZKVKLLBSYIB5NNVSW6W2P3ZGB32767JQD3IAQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.430681303 (475h32m50.301764348s) [noise=true , address=145.79.227.5:4222 , piece_id=4LE6IHY6PEYFLBXFWRFLYT4PW2HEZ7EC36WLRWKSZD6JJTYBMUWA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.651681582 (475h32m50.522764627s) [noise=true , address=77.90.31.18:4808 , piece_id=HHQ2UCY7ZEXJY3PYADPOE3OB5437XTRCHBHQVPIC5HVTQURWH3PQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.670832920 (475h32m50.541915955s) [noise=true , address=77.90.31.19:4844 , piece_id=3BZ7ECWEVLGWB2Y2LCJKDOZ3CXFDCIZXFOYFFEA7HH5545OUW23A, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.689160839 (475h32m50.560243914s) [noise=true , address=45.91.116.11:4120 , piece_id=2LDWGLEITCRFVF36T3NQQEWOSPXO7EI5VTDBYIMAZCGENVY44TJA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.703849528 (475h32m50.574932574s) [noise=true , address=77.90.31.14:4661 , piece_id=CO6CEUAJ7GG5AD2QKR2UYPMCBQ2KZY5FHYKF3E3HQ55PGOXQY2CQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: <nil> Dec 25 11:01:50.703873904 (475h32m50.574956939s) [seg_pos={0 0}, upload=417414]: Segment reached optimal threshold of 13 pieces. Dec 25 11:01:50.704062734 (475h32m50.575145769s) [noise=true , address=45.91.116.13:4156 , piece_id=XUNA3KXLLUURB5CTQYUNBGFM3E2EVAPPJXODHGXSA4TUOGSYOIRQ, seg_pos={0 0}, upload=417414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 11:01:50.704070509 (475h32m50.575153554s) [noise=true , address=45.91.116.13:4156 , piece_id=XUNA3KXLLUURB5CTQYUNBGFM3E2EVAPPJXODHGXSA4TUOGSYOIRQ, seg_pos={0 0}, upload=417414]: Long tail cancellation due to optimal threshold Dec 25 11:01:50.704283895 (475h32m50.57536694s) [noise=true , address=77.90.31.21:4878 , piece_id=5XE2A4QEQT4HH3MWH4KAND357N2Z5KW2TU4GPR4UHTOUW5QZGX7A, seg_pos={0 0}, upload=417414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 25 11:01:50.704290407 (475h32m50.575373442s) [noise=true , address=77.90.31.21:4878 , piece_id=5XE2A4QEQT4HH3MWH4KAND357N2Z5KW2TU4GPR4UHTOUW5QZGX7A, seg_pos={0 0}, upload=417414]: Long tail cancellation due to optimal threshold Dec 25 11:01:50.704385759 (475h32m50.575468804s) [noise=true , address=45.91.116.8:4048 , piece_id=4CXNEPTHDTX6DUJIDK5G724FL5SVMCU3J6NMOEUSPKSZEMG5DKQA, seg_pos={0 0}, upload=417414]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 25 11:01:50.704408743 (475h32m50.575491788s) [noise=true , address=45.91.116.8:4048 , piece_id=4CXNEPTHDTX6DUJIDK5G724FL5SVMCU3J6NMOEUSPKSZEMG5DKQA, seg_pos={0 0}, upload=417414]: Long tail cancellation due to optimal threshold Dec 25 11:01:50.704439111 (475h32m50.575522146s) [seg_pos={0 0}, upload=417414]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:01:50.704550974 (475h32m50.575634009s) [upload=417414]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:01:50.704557035 (475h32m50.57564007s) [upload=417414]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:01:51.002359738 (475h32m50.873442803s) [upload=417414]: Done uploading segments... Dec 25 11:01:51.002384385 (475h32m50.87346743s) [upload=417414]: Upload finished. err: <nil> Duration: 30.005087954s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.......................X...........XXX...................X] duration: 30.003076745s trace: https://jaeger.dev.storj.io/trace/3947ba1f6020f212 FAIL (2025-12-25 11:01:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: AQUXNC56NRWT3D5LEZV4W2NU577XU3BIPFIQ5BPTCJXYLOS5ARQA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 2JXUPOBAJM45EQFVVDGPUZVIXYBFBGL3CF3KUJGJLCZRMIEPOJDA) context deadline exceeded ======LOGS====== Dec 25 11:00:48.541720060 (475h31m48.412803095s) [upload=417399]: Starting upload Dec 25 11:00:48.541773201 (475h31m48.412856246s) [upload=417399]: Done starting upload Dec 25 11:00:48.541844326 (475h31m48.412927372s) [upload=417399]: Uploading segments... Dec 25 11:00:48.541988762 (475h31m48.413071797s) [upload=417399]: Got next segment. Inline: false Dec 25 11:00:48.542089904 (475h31m48.413172939s) [upload=417399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 11:00:48.542095796 (475h31m48.413178831s) [upload=417399]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 11:00:48.792308358 (475h31m48.663391403s) [seg_pos={0 0}, upload=417399]: Begin upload segment... Dec 25 11:00:48.792368423 (475h31m48.663451458s) [seg_pos={0 0}, upload=417399]: Done begin upload segment. Dec 25 11:00:48.792409110 (475h31m48.663492145s) [noise=true , address=145.79.227.41:4087 , piece_id=ZSDC73NF5UJNI22KIVXPBDHXCYZYP732LWASUNCTZ6HFYVYN3S3A, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.796605160 (475h31m48.667688225s) [noise=true , address=45.91.116.4:4937 , piece_id=RKBVDKZNZZZAIJLZ6DIA3TSSKTFAPHB7LPEULTRHN6GYUJ3CBEKQ, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.798085529 (475h31m48.669168564s) [noise=true , address=145.79.227.4:4185 , piece_id=UMMTFCUAYTFSDAL475VJJLAYJWU6MMD5KEKRKQDSKOLGCT2NO4MQ, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.799247010 (475h31m48.670330045s) [noise=true , address=145.79.227.45:4231 , piece_id=QEPA455AFGIRAIF3U5ES2LUEIMWVVAJZETWNLZ6IN7UCNO6CK25A, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.800435363 (475h31m48.671518398s) [noise=true , address=145.79.227.47:4265 , piece_id=SPTPGTJ3YP6HUF7K2ZD7WJTNNNKYSBMYHUSLQXFFIOQSSTLWXORA, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.801582067 (475h31m48.672665102s) [noise=true , address=45.91.116.10:4084 , piece_id=TYOYQHRYHMMSAEJPOK3IL47HYSLYSPH4W5LQ5FKMHIQNJTF2FBGA, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.802962095 (475h31m48.67404514s) [noise=true , address=77.90.31.15:4700 , piece_id=2JXUPOBAJM45EQFVVDGPUZVIXYBFBGL3CF3KUJGJLCZRMIEPOJDA, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.803983770 (475h31m48.675066815s) [noise=true , address=145.79.227.39:4013 , piece_id=IFRVJTNZPBSE3D4S3RBI6H65HRBCNGVUVB266FTWG3CZQIYD2LRA, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.805297802 (475h31m48.676380847s) [noise=true , address=145.79.227.40:4051 , piece_id=FCU6BVGH5ULKVUY3IENPQJY4QKAOWSVQ42YELW2EZ32BQITIEDLA, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.806108816 (475h31m48.677191851s) [noise=true , address=45.91.116.13:4154 , piece_id=RFX2XKJDVY6VGY6LUT5IB2OOHG5KDUGRX6KGMBJ23LE25TYRKKLA, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.806824820 (475h31m48.677907855s) [noise=true , address=145.79.227.32:4836 , piece_id=K2XZZXAT6GSRVYQCXLTIY7SRJ5GMR7VAJPI7TLZP7CT2NZCC4WOA, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.807299985 (475h31m48.67838302s) [noise=true , address=145.79.227.33:4869 , piece_id=B5VDHFZTXNRWFMNIDVJS3S4BCVSCPGM7HZUUS56VK6GDJOJVZCEA, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.808169049 (475h31m48.679252084s) [noise=true , address=45.91.116.7:4011 , piece_id=YSHLEDQIM2BLHPB4G73KEEODUYOLJZW3ERYNQBFXIKWRINOYVW5A, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.808428965 (475h31m48.67951201s) [noise=true , address=45.91.116.6:4975 , piece_id=AQUXNC56NRWT3D5LEZV4W2NU577XU3BIPFIQ5BPTCJXYLOS5ARQA, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.809709283 (475h31m48.680792318s) [noise=true , address=145.79.227.2:4115 , piece_id=4KA65POYQR67GBKQTAHUJ4JV6VNONAKNCHCZHCKDIZS5XGG7FYAA, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:48.810963371 (475h31m48.682046416s) [noise=true , address=145.79.227.11:4732 , piece_id=3YOINMW44O2CWVA2EL7YJ6MDIQRQBAGFPUQYGF7FMRHMBXZRNM2Q, seg_pos={0 0}, upload=417399]: Uploading piece... Dec 25 11:00:50.468937700 (475h31m50.340020745s) [upload=417399]: Next returned nil segment Dec 25 11:00:50.468952748 (475h31m50.340035793s) [upload=417399]: Waiting for error group managing segments... Dec 25 11:00:50.802658215 (475h31m50.67374126s) [noise=true , address=145.79.227.4:4185 , piece_id=UMMTFCUAYTFSDAL475VJJLAYJWU6MMD5KEKRKQDSKOLGCT2NO4MQ, seg_pos={0 0}, upload=417399]: Done uploading piece. err: <nil> Dec 25 11:00:50.803708465 (475h31m50.6747915s) [noise=true , address=145.79.227.47:4265 , piece_id=SPTPGTJ3YP6HUF7K2ZD7WJTNNNKYSBMYHUSLQXFFIOQSSTLWXORA, seg_pos={0 0}, upload=417399]: Done uploading piece. err: <nil> Dec 25 11:00:50.806565155 (475h31m50.67764819s) [noise=true , address=145.79.227.45:4231 , piece_id=QEPA455AFGIRAIF3U5ES2LUEIMWVVAJZETWNLZ6IN7UCNO6CK25A, seg_pos={0 0}, upload=417399]: Done uploading piece. err: <nil> Dec 25 11:00:50.811808488 (475h31m50.682891524s) [noise=true , address=145.79.227.41:4087 , piece_id=ZSDC73NF5UJNI22KIVXPBDHXCYZYP732LWASUNCTZ6HFYVYN3S3A, seg_pos={0 0}, upload=417399]: Done uploading piece. err: <nil> Dec 25 11:00:50.812286118 (475h31m50.683369153s) [noise=true , address=145.79.227.39:4013 , piece_id=IFRVJTNZPBSE3D4S3RBI6H65HRBCNGVUVB266FTWG3CZQIYD2LRA, seg_pos={0 0}, upload=417399]: Done uploading piece. err: <nil> Dec 25 11:00:50.816035528 (475h31m50.687118563s) [noise=true , address=145.79.227.40:4051 , piece_id=FCU6BVGH5ULKVUY3IENPQJY4QKAOWSVQ42YELW2EZ32BQITIEDLA, seg_pos={0 0}, upload=417399]: Done uploading piece. err: <nil> Dec 25 11:00:50.816821805 (475h31m50.68790485s) [noise=true , address=145.79.227.11:4732 , piece_id=3YOINMW44O2CWVA2EL7YJ6MDIQRQBAGFPUQYGF7FMRHMBXZRNM2Q, seg_pos={0 0}, upload=417399]: Done uploading piece. err: <nil> Dec 25 11:00:50.818827404 (475h31m50.689910449s) [noise=true , address=145.79.227.32:4836 , piece_id=K2XZZXAT6GSRVYQCXLTIY7SRJ5GMR7VAJPI7TLZP7CT2NZCC4WOA, seg_pos={0 0}, upload=417399]: Done uploading piece. err: <nil> Dec 25 11:00:50.832290008 (475h31m50.703373054s) [noise=true , address=145.79.227.33:4869 , piece_id=B5VDHFZTXNRWFMNIDVJS3S4BCVSCPGM7HZUUS56VK6GDJOJVZCEA, seg_pos={0 0}, upload=417399]: Done uploading piece. err: <nil> Dec 25 11:00:50.840037391 (475h31m50.711120436s) [noise=true , address=145.79.227.2:4115 , piece_id=4KA65POYQR67GBKQTAHUJ4JV6VNONAKNCHCZHCKDIZS5XGG7FYAA, seg_pos={0 0}, upload=417399]: Done uploading piece. err: <nil> Dec 25 11:00:51.059685786 (475h31m50.930768831s) [noise=true , address=77.90.31.15:4700 , piece_id=2JXUPOBAJM45EQFVVDGPUZVIXYBFBGL3CF3KUJGJLCZRMIEPOJDA, seg_pos={0 0}, upload=417399]: Done uploading piece. err: <nil> Dec 25 11:00:51.160681697 (475h31m51.031764732s) [noise=true , address=45.91.116.6:4975 , piece_id=AQUXNC56NRWT3D5LEZV4W2NU577XU3BIPFIQ5BPTCJXYLOS5ARQA, seg_pos={0 0}, upload=417399]: Done uploading piece. err: <nil> Dec 25 11:00:51.188256224 (475h31m51.059339269s) [noise=true , address=45.91.116.4:4937 , piece_id=RKBVDKZNZZZAIJLZ6DIA3TSSKTFAPHB7LPEULTRHN6GYUJ3CBEKQ, seg_pos={0 0}, upload=417399]: Done uploading piece. err: <nil> Dec 25 11:00:51.188278936 (475h31m51.059361971s) [seg_pos={0 0}, upload=417399]: Segment reached optimal threshold of 13 pieces. Dec 25 11:00:51.188492955 (475h31m51.05957602s) [noise=true , address=45.91.116.13:4154 , piece_id=RFX2XKJDVY6VGY6LUT5IB2OOHG5KDUGRX6KGMBJ23LE25TYRKKLA, seg_pos={0 0}, upload=417399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 25 11:00:51.188501240 (475h31m51.059584285s) [noise=true , address=45.91.116.13:4154 , piece_id=RFX2XKJDVY6VGY6LUT5IB2OOHG5KDUGRX6KGMBJ23LE25TYRKKLA, seg_pos={0 0}, upload=417399]: Long tail cancellation due to optimal threshold Dec 25 11:00:51.188625957 (475h31m51.059709002s) [noise=true , address=45.91.116.7:4011 , piece_id=YSHLEDQIM2BLHPB4G73KEEODUYOLJZW3ERYNQBFXIKWRINOYVW5A, seg_pos={0 0}, upload=417399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 25 11:00:51.188633542 (475h31m51.059716588s) [noise=true , address=45.91.116.7:4011 , piece_id=YSHLEDQIM2BLHPB4G73KEEODUYOLJZW3ERYNQBFXIKWRINOYVW5A, seg_pos={0 0}, upload=417399]: Long tail cancellation due to optimal threshold Dec 25 11:00:51.188889049 (475h31m51.059972084s) [noise=true , address=45.91.116.10:4084 , piece_id=TYOYQHRYHMMSAEJPOK3IL47HYSLYSPH4W5LQ5FKMHIQNJTF2FBGA, seg_pos={0 0}, upload=417399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: read tcp 188.166.225.170:60676->45.91.116.10:4084: use of closed network connection; context canceled Dec 25 11:00:51.188914377 (475h31m51.059997412s) [noise=true , address=45.91.116.10:4084 , piece_id=TYOYQHRYHMMSAEJPOK3IL47HYSLYSPH4W5LQ5FKMHIQNJTF2FBGA, seg_pos={0 0}, upload=417399]: Long tail cancellation due to optimal threshold Dec 25 11:00:51.188932691 (475h31m51.060015737s) [seg_pos={0 0}, upload=417399]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 11:00:51.189019016 (475h31m51.060102051s) [upload=417399]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:00:51.189024186 (475h31m51.060107231s) [upload=417399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:00:51.489282154 (475h31m51.360365219s) [upload=417399]: Done uploading segments... Dec 25 11:00:51.489301971 (475h31m51.360385016s) [upload=417399]: Upload finished. err: <nil> Duration: 30.003098747s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.......................X...........XXX...................] duration: 6.099089697s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 11:00:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 6.099118773s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.......................X...........XXX..................] duration: 6.070284461s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:59:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 6.070384832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.......................X...........XXX.................] duration: 5.769192228s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:58:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.769215283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.......................X...........XXX................] duration: 6.070169979s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:58:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 6.070211489s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.......................X...........XXX...............] duration: 5.777329167s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:57:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.777353394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.......................X...........XXX..............] duration: 6.074117525s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:57:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 6.074147272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.......................X...........XXX.............] duration: 5.820608553s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:56:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.820722502s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X.......................X...........XXX............] duration: 5.776970413s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:55:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.777001221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X.......................X...........XXX...........] duration: 5.662595205s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:55:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.66264978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X.......................X...........XXX..........] duration: 6.140431618s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:55:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 6.140456897s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X.......................X...........XXX.........] duration: 5.654930889s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:54:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.654955416s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X.......................X...........XXX........] duration: 5.840515855s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:53:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.840555049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X.......................X...........XXX.......] duration: 5.690798689s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:52:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.690828677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..............X.......................X...........XXX......] duration: 6.034815688s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:51:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 6.034840455s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..............X.......................X...........XXX.....] duration: 5.849736353s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:51:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.849767711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.X..............X.......................X...........XXX....] duration: 5.903625945s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:50:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.903647174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.X..............X.......................X...........XXX...] duration: 5.961459558s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:50:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.961481369s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.X..............X.......................X...........XXX..] duration: 5.802377239s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:49:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.802419018s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.X..............X.......................X...........XXX.] duration: 5.777970418s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:47:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.777998462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.X..............X.......................X...........XXX] duration: 30.003831139s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:47:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 30.003849465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.X..............X.......................X...........XX] duration: 30.002785845s trace: https://jaeger.dev.storj.io/trace/14c6733952a1f673 FAIL (2025-12-25 10:46:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: W3ZCLCVTO3RJUHWXOEGCH4PVRVQR6CDS6NWT3B3X7JODWA3HOU4Q) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: PRJY6FWGVA45XVIB2BYVC5TQWTHOBAR2ES4BSWRG66INGDYROCLA) context deadline exceeded ======LOGS====== Dec 25 10:46:03.645183905 (475h17m3.51626694s) [upload=417185]: Starting upload Dec 25 10:46:03.645242356 (475h17m3.516325401s) [upload=417185]: Done starting upload Dec 25 10:46:03.645350352 (475h17m3.516433397s) [upload=417185]: Uploading segments... Dec 25 10:46:03.645464529 (475h17m3.516547574s) [upload=417185]: Got next segment. Inline: false Dec 25 10:46:03.645513552 (475h17m3.516596607s) [upload=417185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:46:03.645519213 (475h17m3.516602258s) [upload=417185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:46:03.895050569 (475h17m3.766133634s) [seg_pos={0 0}, upload=417185]: Begin upload segment... Dec 25 10:46:03.895109140 (475h17m3.766192185s) [seg_pos={0 0}, upload=417185]: Done begin upload segment. Dec 25 10:46:03.895148155 (475h17m3.7662312s) [noise=true , address=145.79.227.6:4257 , piece_id=BMF3G3UUAEUFERN6MUYVTERRMAVDEHKT43LMC4R675VTYE64MDTA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.895292209 (475h17m3.766375254s) [noise=true , address=77.90.31.10:4517 , piece_id=54HHABZ22DYCZSVSNTPLBBVS6EZ27OCRKHBQJLQNDLXUHHSDAOSQ, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.895967885 (475h17m3.76705093s) [noise=true , address=145.79.227.19:4079 , piece_id=B5J7FJLJ2ZRFID35G57MIJTL6TVI7KFQAKPTQT2DWE4NB7JPPJEQ, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.896279839 (475h17m3.767362884s) [noise=true , address=77.90.31.13:4627 , piece_id=W3ZCLCVTO3RJUHWXOEGCH4PVRVQR6CDS6NWT3B3X7JODWA3HOU4Q, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.896572687 (475h17m3.767655732s) [noise=true , address=145.79.227.51:4410 , piece_id=WJHZKGZI5ZHGWLMTIPJ7IDDTEEHJEWS4RRQIYKAZJSAUS3LSHJWQ, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.896858922 (475h17m3.767941967s) [noise=true , address=145.79.227.30:4762 , piece_id=6QZUNES3O6VDPNZHYWFOPJOAGNVPUOJH6GBAZ7CJCRKPONUK3YDA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.897137353 (475h17m3.768220398s) [noise=true , address=145.79.227.29:4942 , piece_id=ZNQCE5XFLNBPLPLE4RM6JX2EBRFYMU4NME2CIZSIYVSTNJ5BJTQA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.897455238 (475h17m3.768538283s) [noise=true , address=145.79.227.12:4765 , piece_id=GCABNPP5CGX6S4WIM6NP4KMYXDNN7E6M75ADPGRIK37WVOVS2XZA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.899403458 (475h17m3.770486493s) [noise=true , address=145.79.227.7:4294 , piece_id=IFGGXVLBMTXDLVBJKQOZFAYOHUCYGTPUEE66ATS5D4FX3BLOXVFQ, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.900795017 (475h17m3.771878062s) [noise=true , address=145.79.227.48:4302 , piece_id=FP7NBQ2ILY7L7KRQ26NLMBSHTCFLPL2NFU5L5IZXJ3XTL6LHBFZQ, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.902252162 (475h17m3.773335197s) [noise=true , address=77.90.31.11:4554 , piece_id=PRJY6FWGVA45XVIB2BYVC5TQWTHOBAR2ES4BSWRG66INGDYROCLA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.904673953 (475h17m3.775756988s) [noise=true , address=145.79.227.25:4618 , piece_id=DFMBHHPDKZHPFYF73SWN5BB22SYOFQUMJFUWUK65PQL2TUZGALMQ, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.906246407 (475h17m3.777329452s) [noise=true , address=45.91.116.14:4192 , piece_id=BU745DZPWUPHZSQV67KBBVTCIYX3NOSUEUZCONHKLRYF7BUVWEGA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.907591038 (475h17m3.778674073s) [noise=true , address=145.79.227.46:4601 , piece_id=QWPOBUQGQZHBLJTPQ5PFO2TX7BE45BFZ3AWNCHSGCRZAVCZSMMQA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.908854654 (475h17m3.779937699s) [noise=true , address=145.79.227.43:4158 , piece_id=RGMWBHRCVKOAOWO26UOY4HBPN2UAHA3JTACIG6ZP7LKAYWMG4DVA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.910108662 (475h17m3.781191707s) [noise=true , address=145.79.227.47:4265 , piece_id=ZSVY4N2QBE6NHB336L6UGYJKQ4HBHBTOHUHM7WW5WGOP4DSJ5SSQ, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:05.619526557 (475h17m5.490609622s) [upload=417185]: Next returned nil segment Dec 25 10:46:05.619555642 (475h17m5.490638677s) [upload=417185]: Waiting for error group managing segments... Dec 25 10:46:05.909330237 (475h17m5.780413283s) [noise=true , address=145.79.227.25:4618 , piece_id=DFMBHHPDKZHPFYF73SWN5BB22SYOFQUMJFUWUK65PQL2TUZGALMQ, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.916522563 (475h17m5.787605608s) [noise=true , address=145.79.227.46:4601 , piece_id=QWPOBUQGQZHBLJTPQ5PFO2TX7BE45BFZ3AWNCHSGCRZAVCZSMMQA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.922864139 (475h17m5.793947184s) [noise=true , address=145.79.227.51:4410 , piece_id=WJHZKGZI5ZHGWLMTIPJ7IDDTEEHJEWS4RRQIYKAZJSAUS3LSHJWQ, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.925332168 (475h17m5.796415203s) [noise=true , address=145.79.227.48:4302 , piece_id=FP7NBQ2ILY7L7KRQ26NLMBSHTCFLPL2NFU5L5IZXJ3XTL6LHBFZQ, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.929558896 (475h17m5.800641941s) [noise=true , address=145.79.227.19:4079 , piece_id=B5J7FJLJ2ZRFID35G57MIJTL6TVI7KFQAKPTQT2DWE4NB7JPPJEQ, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.934438628 (475h17m5.805521663s) [noise=true , address=145.79.227.12:4765 , piece_id=GCABNPP5CGX6S4WIM6NP4KMYXDNN7E6M75ADPGRIK37WVOVS2XZA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.936452463 (475h17m5.807535498s) [noise=true , address=145.79.227.7:4294 , piece_id=IFGGXVLBMTXDLVBJKQOZFAYOHUCYGTPUEE66ATS5D4FX3BLOXVFQ, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.938350758 (475h17m5.809433803s) [noise=true , address=145.79.227.43:4158 , piece_id=RGMWBHRCVKOAOWO26UOY4HBPN2UAHA3JTACIG6ZP7LKAYWMG4DVA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.941483022 (475h17m5.812566077s) [noise=true , address=145.79.227.29:4942 , piece_id=ZNQCE5XFLNBPLPLE4RM6JX2EBRFYMU4NME2CIZSIYVSTNJ5BJTQA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.962315901 (475h17m5.833398936s) [noise=true , address=145.79.227.47:4265 , piece_id=ZSVY4N2QBE6NHB336L6UGYJKQ4HBHBTOHUHM7WW5WGOP4DSJ5SSQ, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.986971289 (475h17m5.858054334s) [noise=true , address=145.79.227.6:4257 , piece_id=BMF3G3UUAEUFERN6MUYVTERRMAVDEHKT43LMC4R675VTYE64MDTA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:06.100560449 (475h17m5.971643494s) [noise=true , address=77.90.31.13:4627 , piece_id=W3ZCLCVTO3RJUHWXOEGCH4PVRVQR6CDS6NWT3B3X7JODWA3HOU4Q, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:06.173289307 (475h17m6.044372343s) [noise=true , address=77.90.31.11:4554 , piece_id=PRJY6FWGVA45XVIB2BYVC5TQWTHOBAR2ES4BSWRG66INGDYROCLA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:06.173312711 (475h17m6.044395746s) [seg_pos={0 0}, upload=417185]: Segment reached optimal threshold of 13 pieces. Dec 25 10:46:06.173523022 (475h17m6.044606067s) [noise=true , address=145.79.227.30:4762 , piece_id=6QZUNES3O6VDPNZHYWFOPJOAGNVPUOJH6GBAZ7CJCRKPONUK3YDA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 10:46:06.173530988 (475h17m6.044614033s) [noise=true , address=145.79.227.30:4762 , piece_id=6QZUNES3O6VDPNZHYWFOPJOAGNVPUOJH6GBAZ7CJCRKPONUK3YDA, seg_pos={0 0}, upload=417185]: Long tail cancellation due to optimal threshold Dec 25 10:46:06.173618043 (475h17m6.044701078s) [noise=true , address=45.91.116.14:4192 , piece_id=BU745DZPWUPHZSQV67KBBVTCIYX3NOSUEUZCONHKLRYF7BUVWEGA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:58040->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:58040->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58040->45.91.116.14:4192: use of closed network connection Dec 25 10:46:06.173629775 (475h17m6.04471281s) [noise=true , address=45.91.116.14:4192 , piece_id=BU745DZPWUPHZSQV67KBBVTCIYX3NOSUEUZCONHKLRYF7BUVWEGA, seg_pos={0 0}, upload=417185]: Long tail cancellation due to optimal threshold Dec 25 10:46:06.174036830 (475h17m6.045119865s) [noise=true , address=77.90.31.10:4517 , piece_id=54HHABZ22DYCZSVSNTPLBBVS6EZ27OCRKHBQJLQNDLXUHHSDAOSQ, seg_pos={0 0}, upload=417185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:46:06.174044815 (475h17m6.04512786s) [noise=true , address=77.90.31.10:4517 , piece_id=54HHABZ22DYCZSVSNTPLBBVS6EZ27OCRKHBQJLQNDLXUHHSDAOSQ, seg_pos={0 0}, upload=417185]: Long tail cancellation due to optimal threshold Dec 25 10:46:06.174066768 (475h17m6.045149813s) [seg_pos={0 0}, upload=417185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:46:06.174192768 (475h17m6.045275833s) [upload=417185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:46:06.174200823 (475h17m6.045283858s) [upload=417185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:46:06.466033598 (475h17m6.337116633s) [upload=417185]: Done uploading segments... Dec 25 10:46:06.466053907 (475h17m6.337136952s) [upload=417185]: Upload finished. err: <nil> Duration: 30.002809791s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.X..............X.......................X...........X] duration: 30.003685319s trace: https://jaeger.dev.storj.io/trace/5159ecaf1e8f769b FAIL (2025-12-25 10:45:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:45:21.720192346 (475h16m21.591275391s) [upload=417175]: Starting upload Dec 25 10:45:21.720254234 (475h16m21.591337269s) [upload=417175]: Done starting upload Dec 25 10:45:21.720534858 (475h16m21.591617893s) [upload=417175]: Uploading segments... Dec 25 10:45:21.720580615 (475h16m21.59166365s) [upload=417175]: Got next segment. Inline: false Dec 25 10:45:21.720667140 (475h16m21.591750185s) [upload=417175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:45:21.720672040 (475h16m21.591755085s) [upload=417175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:45:21.969061180 (475h16m21.840144225s) [seg_pos={0 0}, upload=417175]: Begin upload segment... Dec 25 10:45:21.969163715 (475h16m21.84024675s) [seg_pos={0 0}, upload=417175]: Done begin upload segment. Dec 25 10:45:21.969219021 (475h16m21.840302066s) [noise=true , address=45.91.116.11:4119 , piece_id=R4QSAYXV72U5OXU3J6U5C5FUC4MDRZAA6DYZDKD3OGUNXRAJCKPA, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.969242807 (475h16m21.840325842s) [noise=true , address=145.79.227.52:4446 , piece_id=MWVPORRBN4YEKYGCPZIDH7WLX7QIWX63WJRPREKPZUTWZC6AELHQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.970182074 (475h16m21.841265119s) [noise=true , address=45.91.116.12:4228 , piece_id=EDNHWBC2DW2XB56N4ALRYTKZCGWXHTHGEDMGZR5F2ZV645FPKZOA, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.970279560 (475h16m21.841362595s) [noise=true , address=145.79.227.48:4303 , piece_id=VJHEP3JVATB34BI2GDE2XE6NSRDK4R2P47C22W7NJCRIB4CGW66A, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.970653412 (475h16m21.841736457s) [noise=true , address=45.91.116.7:4010 , piece_id=R2G5JLUDV4RTB4UBJBCCRLYYBARHB3M6JWS7RFMI3GXC4FSHE6GA, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.970672479 (475h16m21.841755524s) [noise=true , address=45.91.116.14:4191 , piece_id=CKZHSYM36LQUNKJIVSD4QUZHZLDFNVWN24TWUTPHMORUP2PAFGQQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.970994862 (475h16m21.842077908s) [noise=true , address=145.79.227.32:4835 , piece_id=DMC6NUO2W65OBYACK5HQSNIV4CF5TBHS2S7DZG6BEYVFRVKCERFA, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.971040249 (475h16m21.842123294s) [noise=true , address=145.79.227.16:4910 , piece_id=YFGIOIB4LN2TSLUKGII7ADJQKY6NXCPQVLH7OA3BFZZQBSONECMQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.971327696 (475h16m21.842410731s) [noise=true , address=145.79.227.33:4869 , piece_id=SAMY3FPLPJXND6BKGLDH5D74MUKPHVBEM56PF3GJYCB7UGWPRA6Q, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.971511246 (475h16m21.842594281s) [noise=true , address=145.79.227.22:4509 , piece_id=5XI7ETTPCV5S7KOSNLO7M4WZAGSTXRVVNURF3NWIBRMH2RA4AKWQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.971657825 (475h16m21.84274086s) [noise=true , address=45.91.116.5:4297 , piece_id=QTACZCRNIMRRCQ77YDF2XKGZ36KGXMKMYWHJHNT2KZEPBOLNKLTA, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.972037909 (475h16m21.843120954s) [noise=true , address=145.79.227.3:4150 , piece_id=4EL6LO7D2POLENHTQRAQSTBFDT7ZWISYIP5QH6EYKZOLQWWD6YPQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.972433782 (475h16m21.843516817s) [noise=true , address=145.79.227.29:4941 , piece_id=5APUJOOGHBTN2KAA6ZNMDTNSNAP7YNA3PNLBAXKWVVK7TG2RDGAQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.972829236 (475h16m21.843912271s) [noise=true , address=45.91.116.4:4940 , piece_id=24NQTUS5EXHS2UOWKFVHTX6FFS2DPBQCIEM4KR6NARGZXGH2ABGQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.973180825 (475h16m21.84426387s) [noise=true , address=145.79.227.39:4015 , piece_id=PQBSNMUTFEFCWX7VDYB3GAJZP67QOWTQFMFJH6NTFTFAGA62LE3Q, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.973371698 (475h16m21.844454733s) [noise=true , address=145.79.227.13:4804 , piece_id=Q2OHOUWELLXVDJLOV2PK73VRYRM2I4RKRBR3M3XBNK52LDOG2VQQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:23.595258401 (475h16m23.466341436s) [upload=417175]: Next returned nil segment Dec 25 10:45:23.595322834 (475h16m23.466405899s) [upload=417175]: Waiting for error group managing segments... Dec 25 10:45:23.979830797 (475h16m23.850913842s) [noise=true , address=145.79.227.39:4015 , piece_id=PQBSNMUTFEFCWX7VDYB3GAJZP67QOWTQFMFJH6NTFTFAGA62LE3Q, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:23.980192605 (475h16m23.85127565s) [noise=true , address=145.79.227.13:4804 , piece_id=Q2OHOUWELLXVDJLOV2PK73VRYRM2I4RKRBR3M3XBNK52LDOG2VQQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:23.981783164 (475h16m23.852866199s) [noise=true , address=145.79.227.29:4941 , piece_id=5APUJOOGHBTN2KAA6ZNMDTNSNAP7YNA3PNLBAXKWVVK7TG2RDGAQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:23.985386996 (475h16m23.856470041s) [noise=true , address=145.79.227.32:4835 , piece_id=DMC6NUO2W65OBYACK5HQSNIV4CF5TBHS2S7DZG6BEYVFRVKCERFA, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:23.986596369 (475h16m23.857679414s) [noise=true , address=145.79.227.16:4910 , piece_id=YFGIOIB4LN2TSLUKGII7ADJQKY6NXCPQVLH7OA3BFZZQBSONECMQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:23.988116814 (475h16m23.859199849s) [noise=true , address=145.79.227.3:4150 , piece_id=4EL6LO7D2POLENHTQRAQSTBFDT7ZWISYIP5QH6EYKZOLQWWD6YPQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:23.988571389 (475h16m23.859654434s) [noise=true , address=145.79.227.22:4509 , piece_id=5XI7ETTPCV5S7KOSNLO7M4WZAGSTXRVVNURF3NWIBRMH2RA4AKWQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:23.990639247 (475h16m23.861722282s) [noise=true , address=145.79.227.33:4869 , piece_id=SAMY3FPLPJXND6BKGLDH5D74MUKPHVBEM56PF3GJYCB7UGWPRA6Q, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:24.001703456 (475h16m23.872786501s) [noise=true , address=145.79.227.48:4303 , piece_id=VJHEP3JVATB34BI2GDE2XE6NSRDK4R2P47C22W7NJCRIB4CGW66A, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:24.002585755 (475h16m23.87366879s) [noise=true , address=145.79.227.52:4446 , piece_id=MWVPORRBN4YEKYGCPZIDH7WLX7QIWX63WJRPREKPZUTWZC6AELHQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:24.200065527 (475h16m24.071148562s) [noise=true , address=45.91.116.4:4940 , piece_id=24NQTUS5EXHS2UOWKFVHTX6FFS2DPBQCIEM4KR6NARGZXGH2ABGQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:24.327353743 (475h16m24.198436778s) [noise=true , address=45.91.116.11:4119 , piece_id=R4QSAYXV72U5OXU3J6U5C5FUC4MDRZAA6DYZDKD3OGUNXRAJCKPA, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:24.347264416 (475h16m24.218347452s) [noise=true , address=45.91.116.14:4191 , piece_id=CKZHSYM36LQUNKJIVSD4QUZHZLDFNVWN24TWUTPHMORUP2PAFGQQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:24.347283723 (475h16m24.218366758s) [seg_pos={0 0}, upload=417175]: Segment reached optimal threshold of 13 pieces. Dec 25 10:45:24.347464588 (475h16m24.218547623s) [noise=true , address=45.91.116.5:4297 , piece_id=QTACZCRNIMRRCQ77YDF2XKGZ36KGXMKMYWHJHNT2KZEPBOLNKLTA, seg_pos={0 0}, upload=417175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 25 10:45:24.347471040 (475h16m24.218554085s) [noise=true , address=45.91.116.5:4297 , piece_id=QTACZCRNIMRRCQ77YDF2XKGZ36KGXMKMYWHJHNT2KZEPBOLNKLTA, seg_pos={0 0}, upload=417175]: Long tail cancellation due to optimal threshold Dec 25 10:45:24.347684817 (475h16m24.218767862s) [noise=true , address=45.91.116.12:4228 , piece_id=EDNHWBC2DW2XB56N4ALRYTKZCGWXHTHGEDMGZR5F2ZV645FPKZOA, seg_pos={0 0}, upload=417175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:45:24.347693233 (475h16m24.218776268s) [noise=true , address=45.91.116.12:4228 , piece_id=EDNHWBC2DW2XB56N4ALRYTKZCGWXHTHGEDMGZR5F2ZV645FPKZOA, seg_pos={0 0}, upload=417175]: Long tail cancellation due to optimal threshold Dec 25 10:45:24.347752887 (475h16m24.218835922s) [noise=true , address=45.91.116.7:4010 , piece_id=R2G5JLUDV4RTB4UBJBCCRLYYBARHB3M6JWS7RFMI3GXC4FSHE6GA, seg_pos={0 0}, upload=417175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 25 10:45:24.347767163 (475h16m24.218850198s) [noise=true , address=45.91.116.7:4010 , piece_id=R2G5JLUDV4RTB4UBJBCCRLYYBARHB3M6JWS7RFMI3GXC4FSHE6GA, seg_pos={0 0}, upload=417175]: Long tail cancellation due to optimal threshold Dec 25 10:45:24.347804374 (475h16m24.218887409s) [seg_pos={0 0}, upload=417175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:45:24.347996090 (475h16m24.219079125s) [upload=417175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:45:24.348002842 (475h16m24.219085877s) [upload=417175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:45:24.634659120 (475h16m24.505742165s) [upload=417175]: Done uploading segments... Dec 25 10:45:24.634682264 (475h16m24.505765309s) [upload=417175]: Upload finished. err: <nil> Duration: 30.00372248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.X..............X.......................X...........] duration: 5.777986207s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:44:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.778012707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.X..............X.......................X..........] duration: 5.980897753s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:43:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.980915987s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.X..............X.......................X.........] duration: 5.664987269s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:43:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.665017687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.X..............X.......................X........] duration: 6.051948923s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:43:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 6.051979962s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.X..............X.......................X.......] duration: 6.045380036s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:41:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 6.045409994s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.X..............X.......................X......] duration: 5.767112966s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:41:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.7671407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.X..............X.......................X.....] duration: 5.705655559s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:40:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.705683782s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X.X..............X.......................X....] duration: 5.814839174s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:39:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.81486435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.X..............X.......................X...] duration: 5.900979007s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:38:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.901007301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X.X..............X.......................X..] duration: 5.814040349s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:37:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.814064856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X.X..............X.......................X.] duration: 27.955808766s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:37:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 27.955926611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X.X..............X.......................X] duration: 30.002749438s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:36:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 30.002767552s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.X..............X.......................] duration: 5.694226556s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:35:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.694303563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.X..............X......................] duration: 5.994075634s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:35:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.994107544s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.X..............X.....................] duration: 5.688924322s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:34:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.688953376s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)