Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............................X...X...X..XXX......X.X....X..] duration: 8.55204722s FAIL (2025-10-09 12:44:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: ABGLHL7KB5GAQA3Q457K6KGXTL35GHUBHH4OGGJFNUGBJHTCCLVQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 6ZBKURJZUCPLNPWIRULQQG3FGC2B7CWXDS4DDINNLKDGCPXGBI4Q) context deadline exceeded ======LOGS====== Oct 9 12:42:22.999145300 (383h11m8.461237783s) [upload=337151]: Starting upload Oct 9 12:42:22.999249097 (383h11m8.46134158s) [upload=337151]: Done starting upload Oct 9 12:42:22.999382341 (383h11m8.461474834s) [upload=337151]: Uploading segments... Oct 9 12:42:22.999599855 (383h11m8.461692368s) [upload=337151]: Got next segment. Inline: false Oct 9 12:42:22.999650993 (383h11m8.461743476s) [upload=337151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:42:22.999659429 (383h11m8.461751912s) [upload=337151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:42:23.257741240 (383h11m8.719833733s) [seg_pos={0 0}, upload=337151]: Begin upload segment... Oct 9 12:42:23.257840750 (383h11m8.719933233s) [seg_pos={0 0}, upload=337151]: Done begin upload segment. Oct 9 12:42:23.257852162 (383h11m8.719944645s) [noise=true , address=77.90.31.13:4626 , piece_id=R4JJCNTMY5JPKXB425EIS64JJNV2PQY22JG7HFAMNKCG3TM4NRUQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.257868343 (383h11m8.719960826s) [noise=true , address=145.79.227.12:4767 , piece_id=HBHL5HZ7NY4DFFLT3WYQGKQSUZUGMJ4FROCUZFTSFB6U6P3KFKHA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.258385838 (383h11m8.720478321s) [noise=true , address=77.90.31.21:4877 , piece_id=CGERKUM2XHLE3WIGWJGFEDSQ4ZA3WYWV4ZKV2RY3PR4BYMQSNROA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.258788465 (383h11m8.720880948s) [noise=true , address=145.79.227.7:4295 , piece_id=7B5A22SHYOBVTAPF62LV4EJTZDZQHWHDA672H5BQA3L6ZB7XMKEQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.258932469 (383h11m8.721024953s) [noise=true , address=145.79.227.14:4837 , piece_id=TX76UHILYNK2VF3NB3UBIMUIO4XFCWX3J7F346NDFT2A4AXQJCWQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.259170683 (383h11m8.721263166s) [noise=true , address=145.79.227.4:4187 , piece_id=R47DEKY6TMYY6EEFWJYFNB3KUOBEDNBHFCJHYEEH5WGQ7SYNTZJA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.259310740 (383h11m8.721403253s) [noise=true , address=145.79.227.5:4222 , piece_id=UCXV5U5HEOXTZ3XMZ2DUOFWZF7H2XHIZ3Q5QHKKY47ZTB5EL33IQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.259559373 (383h11m8.721651856s) [noise=true , address=145.79.227.16:4912 , piece_id=OO4E64636DXBNJUBEJQ5X3Z6TVWKVRUCS7P6HAQQW23C466M2RPA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.259669503 (383h11m8.721761986s) [noise=true , address=145.79.227.19:4078 , piece_id=LGMV7RQR5HMADQ3ITDVTSCR44REZJATYHQF3OBOGW2K6OR7YLQQQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.260038094 (383h11m8.722130577s) [noise=true , address=145.79.227.4:4185 , piece_id=Q5ED26RFDUJA7KK4Z4FQ6WSEDB5QW5OCRELULOSMZUKUSNGUCX7Q, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.260966923 (383h11m8.723059396s) [noise=true , address=77.90.31.10:4518 , piece_id=ABGLHL7KB5GAQA3Q457K6KGXTL35GHUBHH4OGGJFNUGBJHTCCLVQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.261334664 (383h11m8.723427147s) [noise=true , address=45.91.116.13:4154 , piece_id=GH5NOPDKCVBS34IMR5ZN7H3OYU6PGCEIZLS73J6NPY4W2NK5PWRA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.261528382 (383h11m8.723620865s) [noise=true , address=77.90.31.10:4517 , piece_id=6ZBKURJZUCPLNPWIRULQQG3FGC2B7CWXDS4DDINNLKDGCPXGBI4Q, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.262006222 (383h11m8.724098715s) [noise=true , address=45.91.116.5:4297 , piece_id=C55J2WSOXP2M533RTYQCQ3GEP7DZXEOD2VJ3S6HHJAYSRKHWULPQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.262416323 (383h11m8.724508806s) [noise=true , address=145.79.227.17:4971 , piece_id=QTWFY6WODFSGIKAXEB6OE4HCXFM4MATIE76GFN3OSAWUI2JEXOJQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.262570456 (383h11m8.724662939s) [noise=true , address=145.79.227.11:4729 , piece_id=EDCQFGE35RAKRRS5AD76VNF55XIOJVZX2MT3BI3TLPH5SF7ACLMQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:25.253713760 (383h11m10.715806233s) [upload=337151]: Next returned nil segment Oct 9 12:42:25.253734398 (383h11m10.715826881s) [upload=337151]: Waiting for error group managing segments... Oct 9 12:42:25.575237677 (383h11m11.03733016s) [noise=true , address=77.90.31.10:4518 , piece_id=ABGLHL7KB5GAQA3Q457K6KGXTL35GHUBHH4OGGJFNUGBJHTCCLVQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.575280088 (383h11m11.037372581s) [noise=true , address=45.91.116.13:4154 , piece_id=GH5NOPDKCVBS34IMR5ZN7H3OYU6PGCEIZLS73J6NPY4W2NK5PWRA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.642114465 (383h11m11.104206948s) [noise=true , address=77.90.31.21:4877 , piece_id=CGERKUM2XHLE3WIGWJGFEDSQ4ZA3WYWV4ZKV2RY3PR4BYMQSNROA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.831085876 (383h11m11.293178359s) [noise=true , address=77.90.31.13:4626 , piece_id=R4JJCNTMY5JPKXB425EIS64JJNV2PQY22JG7HFAMNKCG3TM4NRUQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.956090581 (383h11m11.418183054s) [noise=true , address=77.90.31.10:4517 , piece_id=6ZBKURJZUCPLNPWIRULQQG3FGC2B7CWXDS4DDINNLKDGCPXGBI4Q, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.984612433 (383h11m11.446704917s) [noise=true , address=145.79.227.7:4295 , piece_id=7B5A22SHYOBVTAPF62LV4EJTZDZQHWHDA672H5BQA3L6ZB7XMKEQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.020085632 (383h11m11.482178115s) [noise=true , address=145.79.227.19:4078 , piece_id=LGMV7RQR5HMADQ3ITDVTSCR44REZJATYHQF3OBOGW2K6OR7YLQQQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.021210393 (383h11m11.483302876s) [noise=true , address=145.79.227.5:4222 , piece_id=UCXV5U5HEOXTZ3XMZ2DUOFWZF7H2XHIZ3Q5QHKKY47ZTB5EL33IQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.038113321 (383h11m11.500205794s) [noise=true , address=145.79.227.16:4912 , piece_id=OO4E64636DXBNJUBEJQ5X3Z6TVWKVRUCS7P6HAQQW23C466M2RPA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.065616001 (383h11m11.527708494s) [noise=true , address=145.79.227.14:4837 , piece_id=TX76UHILYNK2VF3NB3UBIMUIO4XFCWX3J7F346NDFT2A4AXQJCWQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.072085871 (383h11m11.534178354s) [noise=true , address=145.79.227.11:4729 , piece_id=EDCQFGE35RAKRRS5AD76VNF55XIOJVZX2MT3BI3TLPH5SF7ACLMQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.072275321 (383h11m11.534367805s) [noise=true , address=145.79.227.4:4185 , piece_id=Q5ED26RFDUJA7KK4Z4FQ6WSEDB5QW5OCRELULOSMZUKUSNGUCX7Q, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.073318578 (383h11m11.535411061s) [noise=true , address=145.79.227.17:4971 , piece_id=QTWFY6WODFSGIKAXEB6OE4HCXFM4MATIE76GFN3OSAWUI2JEXOJQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.073344888 (383h11m11.535437371s) [seg_pos={0 0}, upload=337151]: Segment reached optimal threshold of 13 pieces. Oct 9 12:42:26.073493181 (383h11m11.535585664s) [noise=true , address=45.91.116.5:4297 , piece_id=C55J2WSOXP2M533RTYQCQ3GEP7DZXEOD2VJ3S6HHJAYSRKHWULPQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Oct 9 12:42:26.073506326 (383h11m11.535598809s) [noise=true , address=45.91.116.5:4297 , piece_id=C55J2WSOXP2M533RTYQCQ3GEP7DZXEOD2VJ3S6HHJAYSRKHWULPQ, seg_pos={0 0}, upload=337151]: Long tail cancellation due to optimal threshold Oct 9 12:42:26.073594764 (383h11m11.535687247s) [noise=true , address=145.79.227.4:4187 , piece_id=R47DEKY6TMYY6EEFWJYFNB3KUOBEDNBHFCJHYEEH5WGQ7SYNTZJA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Oct 9 12:42:26.073617347 (383h11m11.53570983s) [noise=true , address=145.79.227.4:4187 , piece_id=R47DEKY6TMYY6EEFWJYFNB3KUOBEDNBHFCJHYEEH5WGQ7SYNTZJA, seg_pos={0 0}, upload=337151]: Long tail cancellation due to optimal threshold Oct 9 12:42:26.073699674 (383h11m11.535792157s) [noise=true , address=145.79.227.12:4767 , piece_id=HBHL5HZ7NY4DFFLT3WYQGKQSUZUGMJ4FROCUZFTSFB6U6P3KFKHA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 9 12:42:26.073708421 (383h11m11.535800904s) [noise=true , address=145.79.227.12:4767 , piece_id=HBHL5HZ7NY4DFFLT3WYQGKQSUZUGMJ4FROCUZFTSFB6U6P3KFKHA, seg_pos={0 0}, upload=337151]: Long tail cancellation due to optimal threshold Oct 9 12:42:26.073730102 (383h11m11.535822595s) [seg_pos={0 0}, upload=337151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:42:26.073851253 (383h11m11.535943736s) [upload=337151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:42:26.073858356 (383h11m11.535950839s) [upload=337151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:42:26.571281652 (383h11m12.033374135s) [upload=337151]: Done uploading segments... Oct 9 12:42:26.571303594 (383h11m12.033396077s) [upload=337151]: Upload finished. err: <nil> Duration: 8.552122424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...............................X...X...X..XXX......X.X....X.] duration: 25.853679191s FAIL (2025-10-09 12:43:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: ABGLHL7KB5GAQA3Q457K6KGXTL35GHUBHH4OGGJFNUGBJHTCCLVQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 6ZBKURJZUCPLNPWIRULQQG3FGC2B7CWXDS4DDINNLKDGCPXGBI4Q) context deadline exceeded ======LOGS====== Oct 9 12:42:22.999145300 (383h11m8.461237783s) [upload=337151]: Starting upload Oct 9 12:42:22.999249097 (383h11m8.46134158s) [upload=337151]: Done starting upload Oct 9 12:42:22.999382341 (383h11m8.461474834s) [upload=337151]: Uploading segments... Oct 9 12:42:22.999599855 (383h11m8.461692368s) [upload=337151]: Got next segment. Inline: false Oct 9 12:42:22.999650993 (383h11m8.461743476s) [upload=337151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:42:22.999659429 (383h11m8.461751912s) [upload=337151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:42:23.257741240 (383h11m8.719833733s) [seg_pos={0 0}, upload=337151]: Begin upload segment... Oct 9 12:42:23.257840750 (383h11m8.719933233s) [seg_pos={0 0}, upload=337151]: Done begin upload segment. Oct 9 12:42:23.257852162 (383h11m8.719944645s) [noise=true , address=77.90.31.13:4626 , piece_id=R4JJCNTMY5JPKXB425EIS64JJNV2PQY22JG7HFAMNKCG3TM4NRUQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.257868343 (383h11m8.719960826s) [noise=true , address=145.79.227.12:4767 , piece_id=HBHL5HZ7NY4DFFLT3WYQGKQSUZUGMJ4FROCUZFTSFB6U6P3KFKHA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.258385838 (383h11m8.720478321s) [noise=true , address=77.90.31.21:4877 , piece_id=CGERKUM2XHLE3WIGWJGFEDSQ4ZA3WYWV4ZKV2RY3PR4BYMQSNROA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.258788465 (383h11m8.720880948s) [noise=true , address=145.79.227.7:4295 , piece_id=7B5A22SHYOBVTAPF62LV4EJTZDZQHWHDA672H5BQA3L6ZB7XMKEQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.258932469 (383h11m8.721024953s) [noise=true , address=145.79.227.14:4837 , piece_id=TX76UHILYNK2VF3NB3UBIMUIO4XFCWX3J7F346NDFT2A4AXQJCWQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.259170683 (383h11m8.721263166s) [noise=true , address=145.79.227.4:4187 , piece_id=R47DEKY6TMYY6EEFWJYFNB3KUOBEDNBHFCJHYEEH5WGQ7SYNTZJA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.259310740 (383h11m8.721403253s) [noise=true , address=145.79.227.5:4222 , piece_id=UCXV5U5HEOXTZ3XMZ2DUOFWZF7H2XHIZ3Q5QHKKY47ZTB5EL33IQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.259559373 (383h11m8.721651856s) [noise=true , address=145.79.227.16:4912 , piece_id=OO4E64636DXBNJUBEJQ5X3Z6TVWKVRUCS7P6HAQQW23C466M2RPA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.259669503 (383h11m8.721761986s) [noise=true , address=145.79.227.19:4078 , piece_id=LGMV7RQR5HMADQ3ITDVTSCR44REZJATYHQF3OBOGW2K6OR7YLQQQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.260038094 (383h11m8.722130577s) [noise=true , address=145.79.227.4:4185 , piece_id=Q5ED26RFDUJA7KK4Z4FQ6WSEDB5QW5OCRELULOSMZUKUSNGUCX7Q, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.260966923 (383h11m8.723059396s) [noise=true , address=77.90.31.10:4518 , piece_id=ABGLHL7KB5GAQA3Q457K6KGXTL35GHUBHH4OGGJFNUGBJHTCCLVQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.261334664 (383h11m8.723427147s) [noise=true , address=45.91.116.13:4154 , piece_id=GH5NOPDKCVBS34IMR5ZN7H3OYU6PGCEIZLS73J6NPY4W2NK5PWRA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.261528382 (383h11m8.723620865s) [noise=true , address=77.90.31.10:4517 , piece_id=6ZBKURJZUCPLNPWIRULQQG3FGC2B7CWXDS4DDINNLKDGCPXGBI4Q, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.262006222 (383h11m8.724098715s) [noise=true , address=45.91.116.5:4297 , piece_id=C55J2WSOXP2M533RTYQCQ3GEP7DZXEOD2VJ3S6HHJAYSRKHWULPQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.262416323 (383h11m8.724508806s) [noise=true , address=145.79.227.17:4971 , piece_id=QTWFY6WODFSGIKAXEB6OE4HCXFM4MATIE76GFN3OSAWUI2JEXOJQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.262570456 (383h11m8.724662939s) [noise=true , address=145.79.227.11:4729 , piece_id=EDCQFGE35RAKRRS5AD76VNF55XIOJVZX2MT3BI3TLPH5SF7ACLMQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:25.253713760 (383h11m10.715806233s) [upload=337151]: Next returned nil segment Oct 9 12:42:25.253734398 (383h11m10.715826881s) [upload=337151]: Waiting for error group managing segments... Oct 9 12:42:25.575237677 (383h11m11.03733016s) [noise=true , address=77.90.31.10:4518 , piece_id=ABGLHL7KB5GAQA3Q457K6KGXTL35GHUBHH4OGGJFNUGBJHTCCLVQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.575280088 (383h11m11.037372581s) [noise=true , address=45.91.116.13:4154 , piece_id=GH5NOPDKCVBS34IMR5ZN7H3OYU6PGCEIZLS73J6NPY4W2NK5PWRA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.642114465 (383h11m11.104206948s) [noise=true , address=77.90.31.21:4877 , piece_id=CGERKUM2XHLE3WIGWJGFEDSQ4ZA3WYWV4ZKV2RY3PR4BYMQSNROA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.831085876 (383h11m11.293178359s) [noise=true , address=77.90.31.13:4626 , piece_id=R4JJCNTMY5JPKXB425EIS64JJNV2PQY22JG7HFAMNKCG3TM4NRUQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.956090581 (383h11m11.418183054s) [noise=true , address=77.90.31.10:4517 , piece_id=6ZBKURJZUCPLNPWIRULQQG3FGC2B7CWXDS4DDINNLKDGCPXGBI4Q, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.984612433 (383h11m11.446704917s) [noise=true , address=145.79.227.7:4295 , piece_id=7B5A22SHYOBVTAPF62LV4EJTZDZQHWHDA672H5BQA3L6ZB7XMKEQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.020085632 (383h11m11.482178115s) [noise=true , address=145.79.227.19:4078 , piece_id=LGMV7RQR5HMADQ3ITDVTSCR44REZJATYHQF3OBOGW2K6OR7YLQQQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.021210393 (383h11m11.483302876s) [noise=true , address=145.79.227.5:4222 , piece_id=UCXV5U5HEOXTZ3XMZ2DUOFWZF7H2XHIZ3Q5QHKKY47ZTB5EL33IQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.038113321 (383h11m11.500205794s) [noise=true , address=145.79.227.16:4912 , piece_id=OO4E64636DXBNJUBEJQ5X3Z6TVWKVRUCS7P6HAQQW23C466M2RPA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.065616001 (383h11m11.527708494s) [noise=true , address=145.79.227.14:4837 , piece_id=TX76UHILYNK2VF3NB3UBIMUIO4XFCWX3J7F346NDFT2A4AXQJCWQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.072085871 (383h11m11.534178354s) [noise=true , address=145.79.227.11:4729 , piece_id=EDCQFGE35RAKRRS5AD76VNF55XIOJVZX2MT3BI3TLPH5SF7ACLMQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.072275321 (383h11m11.534367805s) [noise=true , address=145.79.227.4:4185 , piece_id=Q5ED26RFDUJA7KK4Z4FQ6WSEDB5QW5OCRELULOSMZUKUSNGUCX7Q, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.073318578 (383h11m11.535411061s) [noise=true , address=145.79.227.17:4971 , piece_id=QTWFY6WODFSGIKAXEB6OE4HCXFM4MATIE76GFN3OSAWUI2JEXOJQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.073344888 (383h11m11.535437371s) [seg_pos={0 0}, upload=337151]: Segment reached optimal threshold of 13 pieces. Oct 9 12:42:26.073493181 (383h11m11.535585664s) [noise=true , address=45.91.116.5:4297 , piece_id=C55J2WSOXP2M533RTYQCQ3GEP7DZXEOD2VJ3S6HHJAYSRKHWULPQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Oct 9 12:42:26.073506326 (383h11m11.535598809s) [noise=true , address=45.91.116.5:4297 , piece_id=C55J2WSOXP2M533RTYQCQ3GEP7DZXEOD2VJ3S6HHJAYSRKHWULPQ, seg_pos={0 0}, upload=337151]: Long tail cancellation due to optimal threshold Oct 9 12:42:26.073594764 (383h11m11.535687247s) [noise=true , address=145.79.227.4:4187 , piece_id=R47DEKY6TMYY6EEFWJYFNB3KUOBEDNBHFCJHYEEH5WGQ7SYNTZJA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Oct 9 12:42:26.073617347 (383h11m11.53570983s) [noise=true , address=145.79.227.4:4187 , piece_id=R47DEKY6TMYY6EEFWJYFNB3KUOBEDNBHFCJHYEEH5WGQ7SYNTZJA, seg_pos={0 0}, upload=337151]: Long tail cancellation due to optimal threshold Oct 9 12:42:26.073699674 (383h11m11.535792157s) [noise=true , address=145.79.227.12:4767 , piece_id=HBHL5HZ7NY4DFFLT3WYQGKQSUZUGMJ4FROCUZFTSFB6U6P3KFKHA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 9 12:42:26.073708421 (383h11m11.535800904s) [noise=true , address=145.79.227.12:4767 , piece_id=HBHL5HZ7NY4DFFLT3WYQGKQSUZUGMJ4FROCUZFTSFB6U6P3KFKHA, seg_pos={0 0}, upload=337151]: Long tail cancellation due to optimal threshold Oct 9 12:42:26.073730102 (383h11m11.535822595s) [seg_pos={0 0}, upload=337151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:42:26.073851253 (383h11m11.535943736s) [upload=337151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:42:26.073858356 (383h11m11.535950839s) [upload=337151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:42:26.571281652 (383h11m12.033374135s) [upload=337151]: Done uploading segments... Oct 9 12:42:26.571303594 (383h11m12.033396077s) [upload=337151]: Upload finished. err: <nil> Duration: 25.85371003s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [................................X...X...X..XXX......X.X....X] duration: 30.003256763s FAIL (2025-10-09 12:42:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: ABGLHL7KB5GAQA3Q457K6KGXTL35GHUBHH4OGGJFNUGBJHTCCLVQ) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 6ZBKURJZUCPLNPWIRULQQG3FGC2B7CWXDS4DDINNLKDGCPXGBI4Q) context deadline exceeded ======LOGS====== Oct 9 12:42:22.999145300 (383h11m8.461237783s) [upload=337151]: Starting upload Oct 9 12:42:22.999249097 (383h11m8.46134158s) [upload=337151]: Done starting upload Oct 9 12:42:22.999382341 (383h11m8.461474834s) [upload=337151]: Uploading segments... Oct 9 12:42:22.999599855 (383h11m8.461692368s) [upload=337151]: Got next segment. Inline: false Oct 9 12:42:22.999650993 (383h11m8.461743476s) [upload=337151]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:42:22.999659429 (383h11m8.461751912s) [upload=337151]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:42:23.257741240 (383h11m8.719833733s) [seg_pos={0 0}, upload=337151]: Begin upload segment... Oct 9 12:42:23.257840750 (383h11m8.719933233s) [seg_pos={0 0}, upload=337151]: Done begin upload segment. Oct 9 12:42:23.257852162 (383h11m8.719944645s) [noise=true , address=77.90.31.13:4626 , piece_id=R4JJCNTMY5JPKXB425EIS64JJNV2PQY22JG7HFAMNKCG3TM4NRUQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.257868343 (383h11m8.719960826s) [noise=true , address=145.79.227.12:4767 , piece_id=HBHL5HZ7NY4DFFLT3WYQGKQSUZUGMJ4FROCUZFTSFB6U6P3KFKHA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.258385838 (383h11m8.720478321s) [noise=true , address=77.90.31.21:4877 , piece_id=CGERKUM2XHLE3WIGWJGFEDSQ4ZA3WYWV4ZKV2RY3PR4BYMQSNROA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.258788465 (383h11m8.720880948s) [noise=true , address=145.79.227.7:4295 , piece_id=7B5A22SHYOBVTAPF62LV4EJTZDZQHWHDA672H5BQA3L6ZB7XMKEQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.258932469 (383h11m8.721024953s) [noise=true , address=145.79.227.14:4837 , piece_id=TX76UHILYNK2VF3NB3UBIMUIO4XFCWX3J7F346NDFT2A4AXQJCWQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.259170683 (383h11m8.721263166s) [noise=true , address=145.79.227.4:4187 , piece_id=R47DEKY6TMYY6EEFWJYFNB3KUOBEDNBHFCJHYEEH5WGQ7SYNTZJA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.259310740 (383h11m8.721403253s) [noise=true , address=145.79.227.5:4222 , piece_id=UCXV5U5HEOXTZ3XMZ2DUOFWZF7H2XHIZ3Q5QHKKY47ZTB5EL33IQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.259559373 (383h11m8.721651856s) [noise=true , address=145.79.227.16:4912 , piece_id=OO4E64636DXBNJUBEJQ5X3Z6TVWKVRUCS7P6HAQQW23C466M2RPA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.259669503 (383h11m8.721761986s) [noise=true , address=145.79.227.19:4078 , piece_id=LGMV7RQR5HMADQ3ITDVTSCR44REZJATYHQF3OBOGW2K6OR7YLQQQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.260038094 (383h11m8.722130577s) [noise=true , address=145.79.227.4:4185 , piece_id=Q5ED26RFDUJA7KK4Z4FQ6WSEDB5QW5OCRELULOSMZUKUSNGUCX7Q, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.260966923 (383h11m8.723059396s) [noise=true , address=77.90.31.10:4518 , piece_id=ABGLHL7KB5GAQA3Q457K6KGXTL35GHUBHH4OGGJFNUGBJHTCCLVQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.261334664 (383h11m8.723427147s) [noise=true , address=45.91.116.13:4154 , piece_id=GH5NOPDKCVBS34IMR5ZN7H3OYU6PGCEIZLS73J6NPY4W2NK5PWRA, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.261528382 (383h11m8.723620865s) [noise=true , address=77.90.31.10:4517 , piece_id=6ZBKURJZUCPLNPWIRULQQG3FGC2B7CWXDS4DDINNLKDGCPXGBI4Q, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.262006222 (383h11m8.724098715s) [noise=true , address=45.91.116.5:4297 , piece_id=C55J2WSOXP2M533RTYQCQ3GEP7DZXEOD2VJ3S6HHJAYSRKHWULPQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.262416323 (383h11m8.724508806s) [noise=true , address=145.79.227.17:4971 , piece_id=QTWFY6WODFSGIKAXEB6OE4HCXFM4MATIE76GFN3OSAWUI2JEXOJQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:23.262570456 (383h11m8.724662939s) [noise=true , address=145.79.227.11:4729 , piece_id=EDCQFGE35RAKRRS5AD76VNF55XIOJVZX2MT3BI3TLPH5SF7ACLMQ, seg_pos={0 0}, upload=337151]: Uploading piece... Oct 9 12:42:25.253713760 (383h11m10.715806233s) [upload=337151]: Next returned nil segment Oct 9 12:42:25.253734398 (383h11m10.715826881s) [upload=337151]: Waiting for error group managing segments... Oct 9 12:42:25.575237677 (383h11m11.03733016s) [noise=true , address=77.90.31.10:4518 , piece_id=ABGLHL7KB5GAQA3Q457K6KGXTL35GHUBHH4OGGJFNUGBJHTCCLVQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.575280088 (383h11m11.037372581s) [noise=true , address=45.91.116.13:4154 , piece_id=GH5NOPDKCVBS34IMR5ZN7H3OYU6PGCEIZLS73J6NPY4W2NK5PWRA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.642114465 (383h11m11.104206948s) [noise=true , address=77.90.31.21:4877 , piece_id=CGERKUM2XHLE3WIGWJGFEDSQ4ZA3WYWV4ZKV2RY3PR4BYMQSNROA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.831085876 (383h11m11.293178359s) [noise=true , address=77.90.31.13:4626 , piece_id=R4JJCNTMY5JPKXB425EIS64JJNV2PQY22JG7HFAMNKCG3TM4NRUQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.956090581 (383h11m11.418183054s) [noise=true , address=77.90.31.10:4517 , piece_id=6ZBKURJZUCPLNPWIRULQQG3FGC2B7CWXDS4DDINNLKDGCPXGBI4Q, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:25.984612433 (383h11m11.446704917s) [noise=true , address=145.79.227.7:4295 , piece_id=7B5A22SHYOBVTAPF62LV4EJTZDZQHWHDA672H5BQA3L6ZB7XMKEQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.020085632 (383h11m11.482178115s) [noise=true , address=145.79.227.19:4078 , piece_id=LGMV7RQR5HMADQ3ITDVTSCR44REZJATYHQF3OBOGW2K6OR7YLQQQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.021210393 (383h11m11.483302876s) [noise=true , address=145.79.227.5:4222 , piece_id=UCXV5U5HEOXTZ3XMZ2DUOFWZF7H2XHIZ3Q5QHKKY47ZTB5EL33IQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.038113321 (383h11m11.500205794s) [noise=true , address=145.79.227.16:4912 , piece_id=OO4E64636DXBNJUBEJQ5X3Z6TVWKVRUCS7P6HAQQW23C466M2RPA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.065616001 (383h11m11.527708494s) [noise=true , address=145.79.227.14:4837 , piece_id=TX76UHILYNK2VF3NB3UBIMUIO4XFCWX3J7F346NDFT2A4AXQJCWQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.072085871 (383h11m11.534178354s) [noise=true , address=145.79.227.11:4729 , piece_id=EDCQFGE35RAKRRS5AD76VNF55XIOJVZX2MT3BI3TLPH5SF7ACLMQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.072275321 (383h11m11.534367805s) [noise=true , address=145.79.227.4:4185 , piece_id=Q5ED26RFDUJA7KK4Z4FQ6WSEDB5QW5OCRELULOSMZUKUSNGUCX7Q, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.073318578 (383h11m11.535411061s) [noise=true , address=145.79.227.17:4971 , piece_id=QTWFY6WODFSGIKAXEB6OE4HCXFM4MATIE76GFN3OSAWUI2JEXOJQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: <nil> Oct 9 12:42:26.073344888 (383h11m11.535437371s) [seg_pos={0 0}, upload=337151]: Segment reached optimal threshold of 13 pieces. Oct 9 12:42:26.073493181 (383h11m11.535585664s) [noise=true , address=45.91.116.5:4297 , piece_id=C55J2WSOXP2M533RTYQCQ3GEP7DZXEOD2VJ3S6HHJAYSRKHWULPQ, seg_pos={0 0}, upload=337151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Oct 9 12:42:26.073506326 (383h11m11.535598809s) [noise=true , address=45.91.116.5:4297 , piece_id=C55J2WSOXP2M533RTYQCQ3GEP7DZXEOD2VJ3S6HHJAYSRKHWULPQ, seg_pos={0 0}, upload=337151]: Long tail cancellation due to optimal threshold Oct 9 12:42:26.073594764 (383h11m11.535687247s) [noise=true , address=145.79.227.4:4187 , piece_id=R47DEKY6TMYY6EEFWJYFNB3KUOBEDNBHFCJHYEEH5WGQ7SYNTZJA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Oct 9 12:42:26.073617347 (383h11m11.53570983s) [noise=true , address=145.79.227.4:4187 , piece_id=R47DEKY6TMYY6EEFWJYFNB3KUOBEDNBHFCJHYEEH5WGQ7SYNTZJA, seg_pos={0 0}, upload=337151]: Long tail cancellation due to optimal threshold Oct 9 12:42:26.073699674 (383h11m11.535792157s) [noise=true , address=145.79.227.12:4767 , piece_id=HBHL5HZ7NY4DFFLT3WYQGKQSUZUGMJ4FROCUZFTSFB6U6P3KFKHA, seg_pos={0 0}, upload=337151]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Oct 9 12:42:26.073708421 (383h11m11.535800904s) [noise=true , address=145.79.227.12:4767 , piece_id=HBHL5HZ7NY4DFFLT3WYQGKQSUZUGMJ4FROCUZFTSFB6U6P3KFKHA, seg_pos={0 0}, upload=337151]: Long tail cancellation due to optimal threshold Oct 9 12:42:26.073730102 (383h11m11.535822595s) [seg_pos={0 0}, upload=337151]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:42:26.073851253 (383h11m11.535943736s) [upload=337151]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:42:26.073858356 (383h11m11.535950839s) [upload=337151]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:42:26.571281652 (383h11m12.033374135s) [upload=337151]: Done uploading segments... Oct 9 12:42:26.571303594 (383h11m12.033396077s) [upload=337151]: Upload finished. err: <nil> Duration: 30.00328147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................................X...X...X..XXX......X.X....] duration: 8.993667711s FAIL (2025-10-09 12:42:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 12:38:52.218563236 (383h7m37.680655719s) [upload=337101]: Starting upload Oct 9 12:38:52.218635634 (383h7m37.680728117s) [upload=337101]: Done starting upload Oct 9 12:38:52.218732439 (383h7m37.680824922s) [upload=337101]: Uploading segments... Oct 9 12:38:52.218878457 (383h7m37.68097094s) [upload=337101]: Got next segment. Inline: false Oct 9 12:38:52.218915958 (383h7m37.681008431s) [upload=337101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:38:52.218920767 (383h7m37.68101325s) [upload=337101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:38:52.512937187 (383h7m37.97502968s) [seg_pos={0 0}, upload=337101]: Begin upload segment... Oct 9 12:38:52.513008152 (383h7m37.975100635s) [seg_pos={0 0}, upload=337101]: Done begin upload segment. Oct 9 12:38:52.513042978 (383h7m37.975135461s) [noise=true , address=145.79.227.13:4804 , piece_id=CJBUUUSFPR2QTBZGUK3G5GF4W7X4VAUNB3DR3LVKPKHEHUFRQ52Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513052757 (383h7m37.97514524s) [noise=true , address=45.91.116.11:4117 , piece_id=P33W7XEHJ4QQ6FUVVJVL7FHXKELPFRHXYZ5KLL4FTYPMPNCLTKKQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513106318 (383h7m37.975198801s) [noise=true , address=45.91.116.8:4047 , piece_id=3RRPBR6M2EDNMJLZVLT4K6JAZS3A2AUZU5ZIHFNFBJAQOOLAZUOA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513579680 (383h7m37.975672163s) [noise=true , address=145.79.227.6:4258 , piece_id=5MZVW7TJ4HFSYX27I5FA4H24AGKOUHAQTBHGHPGGO26BIHOQRM6A, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514034265 (383h7m37.976126748s) [noise=true , address=45.91.116.4:4938 , piece_id=H3IXKEMOL5BL4GWUUVI4KPPXBY567L2MGIGOL5E66UDHLNKXUJUA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514429448 (383h7m37.976521932s) [noise=true , address=45.91.116.14:4189 , piece_id=GPJ4HMPQJHNYCDZ4A42BQ5B67OTFZ4WO3DQXRPUP734HMDIMIS6A, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514880015 (383h7m37.976972499s) [noise=true , address=145.79.227.5:4223 , piece_id=65Q5AJCKZTLXGU7DNHQ2GSJJL2CC2KHHJS2VEWKOQKMG2JEHNJ5Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.515321266 (383h7m37.977413749s) [noise=true , address=77.90.31.15:4698 , piece_id=FMCX2XNZ77M7KLAY2SFSFZ4HNYQV6VKPA3LRJXOY4F7H6AUSTNLQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.515771233 (383h7m37.977863716s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.516919299 (383h7m37.979011782s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.517471871 (383h7m37.979564354s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518002952 (383h7m37.980095435s) [noise=true , address=45.91.116.9:4262 , piece_id=EWIPZJTME7DCDFYK3H4IWINR34MXLVVPNPHDLCI5VO3D5IFPQHIA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518550345 (383h7m37.980642828s) [noise=true , address=45.91.116.7:4011 , piece_id=VLDFHTMZVVCM76YBWW5ZAMMUGLJEZ7KLVP6EKZHELGNFYGKEZZMA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518697595 (383h7m37.980790078s) [noise=true , address=145.79.227.2:4115 , piece_id=NNHLVVAJLLRXZD7ZGLBHNWABAVZQPUU5WA65VNUZAUOLZU7SLEMA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.519080835 (383h7m37.981173318s) [noise=true , address=145.79.227.15:4874 , piece_id=RHHDISFENGWL4ODTENVZ7Q6EWTXY4EFFVBUIZ5QPH4B2NXL2U72Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.519577871 (383h7m37.981670354s) [noise=true , address=145.79.227.14:4838 , piece_id=WFDZDAW5GVLX6KZ3FPX2N6JXUCOY2SQOZZZSXKVUWPVAISWEOIRQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:54.383181912 (383h7m39.845274396s) [upload=337101]: Next returned nil segment Oct 9 12:38:54.383204786 (383h7m39.845297269s) [upload=337101]: Waiting for error group managing segments... Oct 9 12:38:54.690706964 (383h7m40.152799447s) [noise=true , address=45.91.116.11:4117 , piece_id=P33W7XEHJ4QQ6FUVVJVL7FHXKELPFRHXYZ5KLL4FTYPMPNCLTKKQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.741232556 (383h7m40.203325039s) [noise=true , address=45.91.116.7:4011 , piece_id=VLDFHTMZVVCM76YBWW5ZAMMUGLJEZ7KLVP6EKZHELGNFYGKEZZMA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.751247917 (383h7m40.2133404s) [noise=true , address=77.90.31.15:4698 , piece_id=FMCX2XNZ77M7KLAY2SFSFZ4HNYQV6VKPA3LRJXOY4F7H6AUSTNLQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.761663620 (383h7m40.223756103s) [noise=true , address=45.91.116.14:4189 , piece_id=GPJ4HMPQJHNYCDZ4A42BQ5B67OTFZ4WO3DQXRPUP734HMDIMIS6A, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.948001437 (383h7m40.41009391s) [noise=true , address=45.91.116.9:4262 , piece_id=EWIPZJTME7DCDFYK3H4IWINR34MXLVVPNPHDLCI5VO3D5IFPQHIA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.138744312 (383h7m40.600836795s) [noise=true , address=45.91.116.4:4938 , piece_id=H3IXKEMOL5BL4GWUUVI4KPPXBY567L2MGIGOL5E66UDHLNKXUJUA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.190920004 (383h7m40.653012488s) [noise=true , address=145.79.227.14:4838 , piece_id=WFDZDAW5GVLX6KZ3FPX2N6JXUCOY2SQOZZZSXKVUWPVAISWEOIRQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.207820096 (383h7m40.669912579s) [noise=true , address=145.79.227.15:4874 , piece_id=RHHDISFENGWL4ODTENVZ7Q6EWTXY4EFFVBUIZ5QPH4B2NXL2U72Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.208620800 (383h7m40.670713283s) [noise=true , address=145.79.227.13:4804 , piece_id=CJBUUUSFPR2QTBZGUK3G5GF4W7X4VAUNB3DR3LVKPKHEHUFRQ52Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.209164125 (383h7m40.671256598s) [noise=true , address=45.91.116.8:4047 , piece_id=3RRPBR6M2EDNMJLZVLT4K6JAZS3A2AUZU5ZIHFNFBJAQOOLAZUOA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.226397842 (383h7m40.688490315s) [noise=true , address=145.79.227.5:4223 , piece_id=65Q5AJCKZTLXGU7DNHQ2GSJJL2CC2KHHJS2VEWKOQKMG2JEHNJ5Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.226930757 (383h7m40.68902324s) [noise=true , address=145.79.227.6:4258 , piece_id=5MZVW7TJ4HFSYX27I5FA4H24AGKOUHAQTBHGHPGGO26BIHOQRM6A, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.278102458 (383h7m40.740194931s) [noise=true , address=145.79.227.2:4115 , piece_id=NNHLVVAJLLRXZD7ZGLBHNWABAVZQPUU5WA65VNUZAUOLZU7SLEMA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.278138768 (383h7m40.740231241s) [seg_pos={0 0}, upload=337101]: Segment reached optimal threshold of 13 pieces. Oct 9 12:38:55.278291768 (383h7m40.740384252s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Oct 9 12:38:55.278301257 (383h7m40.74039374s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278385878 (383h7m40.740478361s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 9 12:38:55.278392911 (383h7m40.740485394s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278499424 (383h7m40.740591907s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection Oct 9 12:38:55.278506477 (383h7m40.74059895s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278525063 (383h7m40.740617546s) [seg_pos={0 0}, upload=337101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:38:55.278609443 (383h7m40.740701926s) [upload=337101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:38:55.278615215 (383h7m40.740707698s) [upload=337101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:38:55.655420291 (383h7m41.117512774s) [upload=337101]: Done uploading segments... Oct 9 12:38:55.655444056 (383h7m41.117536539s) [upload=337101]: Upload finished. err: <nil> Duration: 8.99372499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................................X...X...X..XXX......X.X...] duration: 28.508371794s FAIL (2025-10-09 12:41:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 12:38:52.218563236 (383h7m37.680655719s) [upload=337101]: Starting upload Oct 9 12:38:52.218635634 (383h7m37.680728117s) [upload=337101]: Done starting upload Oct 9 12:38:52.218732439 (383h7m37.680824922s) [upload=337101]: Uploading segments... Oct 9 12:38:52.218878457 (383h7m37.68097094s) [upload=337101]: Got next segment. Inline: false Oct 9 12:38:52.218915958 (383h7m37.681008431s) [upload=337101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:38:52.218920767 (383h7m37.68101325s) [upload=337101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:38:52.512937187 (383h7m37.97502968s) [seg_pos={0 0}, upload=337101]: Begin upload segment... Oct 9 12:38:52.513008152 (383h7m37.975100635s) [seg_pos={0 0}, upload=337101]: Done begin upload segment. Oct 9 12:38:52.513042978 (383h7m37.975135461s) [noise=true , address=145.79.227.13:4804 , piece_id=CJBUUUSFPR2QTBZGUK3G5GF4W7X4VAUNB3DR3LVKPKHEHUFRQ52Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513052757 (383h7m37.97514524s) [noise=true , address=45.91.116.11:4117 , piece_id=P33W7XEHJ4QQ6FUVVJVL7FHXKELPFRHXYZ5KLL4FTYPMPNCLTKKQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513106318 (383h7m37.975198801s) [noise=true , address=45.91.116.8:4047 , piece_id=3RRPBR6M2EDNMJLZVLT4K6JAZS3A2AUZU5ZIHFNFBJAQOOLAZUOA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513579680 (383h7m37.975672163s) [noise=true , address=145.79.227.6:4258 , piece_id=5MZVW7TJ4HFSYX27I5FA4H24AGKOUHAQTBHGHPGGO26BIHOQRM6A, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514034265 (383h7m37.976126748s) [noise=true , address=45.91.116.4:4938 , piece_id=H3IXKEMOL5BL4GWUUVI4KPPXBY567L2MGIGOL5E66UDHLNKXUJUA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514429448 (383h7m37.976521932s) [noise=true , address=45.91.116.14:4189 , piece_id=GPJ4HMPQJHNYCDZ4A42BQ5B67OTFZ4WO3DQXRPUP734HMDIMIS6A, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514880015 (383h7m37.976972499s) [noise=true , address=145.79.227.5:4223 , piece_id=65Q5AJCKZTLXGU7DNHQ2GSJJL2CC2KHHJS2VEWKOQKMG2JEHNJ5Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.515321266 (383h7m37.977413749s) [noise=true , address=77.90.31.15:4698 , piece_id=FMCX2XNZ77M7KLAY2SFSFZ4HNYQV6VKPA3LRJXOY4F7H6AUSTNLQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.515771233 (383h7m37.977863716s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.516919299 (383h7m37.979011782s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.517471871 (383h7m37.979564354s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518002952 (383h7m37.980095435s) [noise=true , address=45.91.116.9:4262 , piece_id=EWIPZJTME7DCDFYK3H4IWINR34MXLVVPNPHDLCI5VO3D5IFPQHIA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518550345 (383h7m37.980642828s) [noise=true , address=45.91.116.7:4011 , piece_id=VLDFHTMZVVCM76YBWW5ZAMMUGLJEZ7KLVP6EKZHELGNFYGKEZZMA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518697595 (383h7m37.980790078s) [noise=true , address=145.79.227.2:4115 , piece_id=NNHLVVAJLLRXZD7ZGLBHNWABAVZQPUU5WA65VNUZAUOLZU7SLEMA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.519080835 (383h7m37.981173318s) [noise=true , address=145.79.227.15:4874 , piece_id=RHHDISFENGWL4ODTENVZ7Q6EWTXY4EFFVBUIZ5QPH4B2NXL2U72Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.519577871 (383h7m37.981670354s) [noise=true , address=145.79.227.14:4838 , piece_id=WFDZDAW5GVLX6KZ3FPX2N6JXUCOY2SQOZZZSXKVUWPVAISWEOIRQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:54.383181912 (383h7m39.845274396s) [upload=337101]: Next returned nil segment Oct 9 12:38:54.383204786 (383h7m39.845297269s) [upload=337101]: Waiting for error group managing segments... Oct 9 12:38:54.690706964 (383h7m40.152799447s) [noise=true , address=45.91.116.11:4117 , piece_id=P33W7XEHJ4QQ6FUVVJVL7FHXKELPFRHXYZ5KLL4FTYPMPNCLTKKQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.741232556 (383h7m40.203325039s) [noise=true , address=45.91.116.7:4011 , piece_id=VLDFHTMZVVCM76YBWW5ZAMMUGLJEZ7KLVP6EKZHELGNFYGKEZZMA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.751247917 (383h7m40.2133404s) [noise=true , address=77.90.31.15:4698 , piece_id=FMCX2XNZ77M7KLAY2SFSFZ4HNYQV6VKPA3LRJXOY4F7H6AUSTNLQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.761663620 (383h7m40.223756103s) [noise=true , address=45.91.116.14:4189 , piece_id=GPJ4HMPQJHNYCDZ4A42BQ5B67OTFZ4WO3DQXRPUP734HMDIMIS6A, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.948001437 (383h7m40.41009391s) [noise=true , address=45.91.116.9:4262 , piece_id=EWIPZJTME7DCDFYK3H4IWINR34MXLVVPNPHDLCI5VO3D5IFPQHIA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.138744312 (383h7m40.600836795s) [noise=true , address=45.91.116.4:4938 , piece_id=H3IXKEMOL5BL4GWUUVI4KPPXBY567L2MGIGOL5E66UDHLNKXUJUA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.190920004 (383h7m40.653012488s) [noise=true , address=145.79.227.14:4838 , piece_id=WFDZDAW5GVLX6KZ3FPX2N6JXUCOY2SQOZZZSXKVUWPVAISWEOIRQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.207820096 (383h7m40.669912579s) [noise=true , address=145.79.227.15:4874 , piece_id=RHHDISFENGWL4ODTENVZ7Q6EWTXY4EFFVBUIZ5QPH4B2NXL2U72Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.208620800 (383h7m40.670713283s) [noise=true , address=145.79.227.13:4804 , piece_id=CJBUUUSFPR2QTBZGUK3G5GF4W7X4VAUNB3DR3LVKPKHEHUFRQ52Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.209164125 (383h7m40.671256598s) [noise=true , address=45.91.116.8:4047 , piece_id=3RRPBR6M2EDNMJLZVLT4K6JAZS3A2AUZU5ZIHFNFBJAQOOLAZUOA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.226397842 (383h7m40.688490315s) [noise=true , address=145.79.227.5:4223 , piece_id=65Q5AJCKZTLXGU7DNHQ2GSJJL2CC2KHHJS2VEWKOQKMG2JEHNJ5Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.226930757 (383h7m40.68902324s) [noise=true , address=145.79.227.6:4258 , piece_id=5MZVW7TJ4HFSYX27I5FA4H24AGKOUHAQTBHGHPGGO26BIHOQRM6A, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.278102458 (383h7m40.740194931s) [noise=true , address=145.79.227.2:4115 , piece_id=NNHLVVAJLLRXZD7ZGLBHNWABAVZQPUU5WA65VNUZAUOLZU7SLEMA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.278138768 (383h7m40.740231241s) [seg_pos={0 0}, upload=337101]: Segment reached optimal threshold of 13 pieces. Oct 9 12:38:55.278291768 (383h7m40.740384252s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Oct 9 12:38:55.278301257 (383h7m40.74039374s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278385878 (383h7m40.740478361s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 9 12:38:55.278392911 (383h7m40.740485394s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278499424 (383h7m40.740591907s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection Oct 9 12:38:55.278506477 (383h7m40.74059895s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278525063 (383h7m40.740617546s) [seg_pos={0 0}, upload=337101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:38:55.278609443 (383h7m40.740701926s) [upload=337101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:38:55.278615215 (383h7m40.740707698s) [upload=337101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:38:55.655420291 (383h7m41.117512774s) [upload=337101]: Done uploading segments... Oct 9 12:38:55.655444056 (383h7m41.117536539s) [upload=337101]: Upload finished. err: <nil> Duration: 28.508398494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................................X...X...X..XXX......X.X..] duration: 16.630171422s FAIL (2025-10-09 12:40:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 12:38:52.218563236 (383h7m37.680655719s) [upload=337101]: Starting upload Oct 9 12:38:52.218635634 (383h7m37.680728117s) [upload=337101]: Done starting upload Oct 9 12:38:52.218732439 (383h7m37.680824922s) [upload=337101]: Uploading segments... Oct 9 12:38:52.218878457 (383h7m37.68097094s) [upload=337101]: Got next segment. Inline: false Oct 9 12:38:52.218915958 (383h7m37.681008431s) [upload=337101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:38:52.218920767 (383h7m37.68101325s) [upload=337101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:38:52.512937187 (383h7m37.97502968s) [seg_pos={0 0}, upload=337101]: Begin upload segment... Oct 9 12:38:52.513008152 (383h7m37.975100635s) [seg_pos={0 0}, upload=337101]: Done begin upload segment. Oct 9 12:38:52.513042978 (383h7m37.975135461s) [noise=true , address=145.79.227.13:4804 , piece_id=CJBUUUSFPR2QTBZGUK3G5GF4W7X4VAUNB3DR3LVKPKHEHUFRQ52Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513052757 (383h7m37.97514524s) [noise=true , address=45.91.116.11:4117 , piece_id=P33W7XEHJ4QQ6FUVVJVL7FHXKELPFRHXYZ5KLL4FTYPMPNCLTKKQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513106318 (383h7m37.975198801s) [noise=true , address=45.91.116.8:4047 , piece_id=3RRPBR6M2EDNMJLZVLT4K6JAZS3A2AUZU5ZIHFNFBJAQOOLAZUOA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513579680 (383h7m37.975672163s) [noise=true , address=145.79.227.6:4258 , piece_id=5MZVW7TJ4HFSYX27I5FA4H24AGKOUHAQTBHGHPGGO26BIHOQRM6A, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514034265 (383h7m37.976126748s) [noise=true , address=45.91.116.4:4938 , piece_id=H3IXKEMOL5BL4GWUUVI4KPPXBY567L2MGIGOL5E66UDHLNKXUJUA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514429448 (383h7m37.976521932s) [noise=true , address=45.91.116.14:4189 , piece_id=GPJ4HMPQJHNYCDZ4A42BQ5B67OTFZ4WO3DQXRPUP734HMDIMIS6A, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514880015 (383h7m37.976972499s) [noise=true , address=145.79.227.5:4223 , piece_id=65Q5AJCKZTLXGU7DNHQ2GSJJL2CC2KHHJS2VEWKOQKMG2JEHNJ5Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.515321266 (383h7m37.977413749s) [noise=true , address=77.90.31.15:4698 , piece_id=FMCX2XNZ77M7KLAY2SFSFZ4HNYQV6VKPA3LRJXOY4F7H6AUSTNLQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.515771233 (383h7m37.977863716s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.516919299 (383h7m37.979011782s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.517471871 (383h7m37.979564354s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518002952 (383h7m37.980095435s) [noise=true , address=45.91.116.9:4262 , piece_id=EWIPZJTME7DCDFYK3H4IWINR34MXLVVPNPHDLCI5VO3D5IFPQHIA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518550345 (383h7m37.980642828s) [noise=true , address=45.91.116.7:4011 , piece_id=VLDFHTMZVVCM76YBWW5ZAMMUGLJEZ7KLVP6EKZHELGNFYGKEZZMA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518697595 (383h7m37.980790078s) [noise=true , address=145.79.227.2:4115 , piece_id=NNHLVVAJLLRXZD7ZGLBHNWABAVZQPUU5WA65VNUZAUOLZU7SLEMA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.519080835 (383h7m37.981173318s) [noise=true , address=145.79.227.15:4874 , piece_id=RHHDISFENGWL4ODTENVZ7Q6EWTXY4EFFVBUIZ5QPH4B2NXL2U72Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.519577871 (383h7m37.981670354s) [noise=true , address=145.79.227.14:4838 , piece_id=WFDZDAW5GVLX6KZ3FPX2N6JXUCOY2SQOZZZSXKVUWPVAISWEOIRQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:54.383181912 (383h7m39.845274396s) [upload=337101]: Next returned nil segment Oct 9 12:38:54.383204786 (383h7m39.845297269s) [upload=337101]: Waiting for error group managing segments... Oct 9 12:38:54.690706964 (383h7m40.152799447s) [noise=true , address=45.91.116.11:4117 , piece_id=P33W7XEHJ4QQ6FUVVJVL7FHXKELPFRHXYZ5KLL4FTYPMPNCLTKKQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.741232556 (383h7m40.203325039s) [noise=true , address=45.91.116.7:4011 , piece_id=VLDFHTMZVVCM76YBWW5ZAMMUGLJEZ7KLVP6EKZHELGNFYGKEZZMA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.751247917 (383h7m40.2133404s) [noise=true , address=77.90.31.15:4698 , piece_id=FMCX2XNZ77M7KLAY2SFSFZ4HNYQV6VKPA3LRJXOY4F7H6AUSTNLQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.761663620 (383h7m40.223756103s) [noise=true , address=45.91.116.14:4189 , piece_id=GPJ4HMPQJHNYCDZ4A42BQ5B67OTFZ4WO3DQXRPUP734HMDIMIS6A, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.948001437 (383h7m40.41009391s) [noise=true , address=45.91.116.9:4262 , piece_id=EWIPZJTME7DCDFYK3H4IWINR34MXLVVPNPHDLCI5VO3D5IFPQHIA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.138744312 (383h7m40.600836795s) [noise=true , address=45.91.116.4:4938 , piece_id=H3IXKEMOL5BL4GWUUVI4KPPXBY567L2MGIGOL5E66UDHLNKXUJUA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.190920004 (383h7m40.653012488s) [noise=true , address=145.79.227.14:4838 , piece_id=WFDZDAW5GVLX6KZ3FPX2N6JXUCOY2SQOZZZSXKVUWPVAISWEOIRQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.207820096 (383h7m40.669912579s) [noise=true , address=145.79.227.15:4874 , piece_id=RHHDISFENGWL4ODTENVZ7Q6EWTXY4EFFVBUIZ5QPH4B2NXL2U72Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.208620800 (383h7m40.670713283s) [noise=true , address=145.79.227.13:4804 , piece_id=CJBUUUSFPR2QTBZGUK3G5GF4W7X4VAUNB3DR3LVKPKHEHUFRQ52Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.209164125 (383h7m40.671256598s) [noise=true , address=45.91.116.8:4047 , piece_id=3RRPBR6M2EDNMJLZVLT4K6JAZS3A2AUZU5ZIHFNFBJAQOOLAZUOA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.226397842 (383h7m40.688490315s) [noise=true , address=145.79.227.5:4223 , piece_id=65Q5AJCKZTLXGU7DNHQ2GSJJL2CC2KHHJS2VEWKOQKMG2JEHNJ5Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.226930757 (383h7m40.68902324s) [noise=true , address=145.79.227.6:4258 , piece_id=5MZVW7TJ4HFSYX27I5FA4H24AGKOUHAQTBHGHPGGO26BIHOQRM6A, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.278102458 (383h7m40.740194931s) [noise=true , address=145.79.227.2:4115 , piece_id=NNHLVVAJLLRXZD7ZGLBHNWABAVZQPUU5WA65VNUZAUOLZU7SLEMA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.278138768 (383h7m40.740231241s) [seg_pos={0 0}, upload=337101]: Segment reached optimal threshold of 13 pieces. Oct 9 12:38:55.278291768 (383h7m40.740384252s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Oct 9 12:38:55.278301257 (383h7m40.74039374s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278385878 (383h7m40.740478361s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 9 12:38:55.278392911 (383h7m40.740485394s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278499424 (383h7m40.740591907s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection Oct 9 12:38:55.278506477 (383h7m40.74059895s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278525063 (383h7m40.740617546s) [seg_pos={0 0}, upload=337101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:38:55.278609443 (383h7m40.740701926s) [upload=337101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:38:55.278615215 (383h7m40.740707698s) [upload=337101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:38:55.655420291 (383h7m41.117512774s) [upload=337101]: Done uploading segments... Oct 9 12:38:55.655444056 (383h7m41.117536539s) [upload=337101]: Upload finished. err: <nil> Duration: 16.630210417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................................X...X...X..XXX......X.X.] duration: 10.471204213s FAIL (2025-10-09 12:39:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 12:38:52.218563236 (383h7m37.680655719s) [upload=337101]: Starting upload Oct 9 12:38:52.218635634 (383h7m37.680728117s) [upload=337101]: Done starting upload Oct 9 12:38:52.218732439 (383h7m37.680824922s) [upload=337101]: Uploading segments... Oct 9 12:38:52.218878457 (383h7m37.68097094s) [upload=337101]: Got next segment. Inline: false Oct 9 12:38:52.218915958 (383h7m37.681008431s) [upload=337101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:38:52.218920767 (383h7m37.68101325s) [upload=337101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:38:52.512937187 (383h7m37.97502968s) [seg_pos={0 0}, upload=337101]: Begin upload segment... Oct 9 12:38:52.513008152 (383h7m37.975100635s) [seg_pos={0 0}, upload=337101]: Done begin upload segment. Oct 9 12:38:52.513042978 (383h7m37.975135461s) [noise=true , address=145.79.227.13:4804 , piece_id=CJBUUUSFPR2QTBZGUK3G5GF4W7X4VAUNB3DR3LVKPKHEHUFRQ52Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513052757 (383h7m37.97514524s) [noise=true , address=45.91.116.11:4117 , piece_id=P33W7XEHJ4QQ6FUVVJVL7FHXKELPFRHXYZ5KLL4FTYPMPNCLTKKQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513106318 (383h7m37.975198801s) [noise=true , address=45.91.116.8:4047 , piece_id=3RRPBR6M2EDNMJLZVLT4K6JAZS3A2AUZU5ZIHFNFBJAQOOLAZUOA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513579680 (383h7m37.975672163s) [noise=true , address=145.79.227.6:4258 , piece_id=5MZVW7TJ4HFSYX27I5FA4H24AGKOUHAQTBHGHPGGO26BIHOQRM6A, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514034265 (383h7m37.976126748s) [noise=true , address=45.91.116.4:4938 , piece_id=H3IXKEMOL5BL4GWUUVI4KPPXBY567L2MGIGOL5E66UDHLNKXUJUA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514429448 (383h7m37.976521932s) [noise=true , address=45.91.116.14:4189 , piece_id=GPJ4HMPQJHNYCDZ4A42BQ5B67OTFZ4WO3DQXRPUP734HMDIMIS6A, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514880015 (383h7m37.976972499s) [noise=true , address=145.79.227.5:4223 , piece_id=65Q5AJCKZTLXGU7DNHQ2GSJJL2CC2KHHJS2VEWKOQKMG2JEHNJ5Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.515321266 (383h7m37.977413749s) [noise=true , address=77.90.31.15:4698 , piece_id=FMCX2XNZ77M7KLAY2SFSFZ4HNYQV6VKPA3LRJXOY4F7H6AUSTNLQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.515771233 (383h7m37.977863716s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.516919299 (383h7m37.979011782s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.517471871 (383h7m37.979564354s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518002952 (383h7m37.980095435s) [noise=true , address=45.91.116.9:4262 , piece_id=EWIPZJTME7DCDFYK3H4IWINR34MXLVVPNPHDLCI5VO3D5IFPQHIA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518550345 (383h7m37.980642828s) [noise=true , address=45.91.116.7:4011 , piece_id=VLDFHTMZVVCM76YBWW5ZAMMUGLJEZ7KLVP6EKZHELGNFYGKEZZMA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518697595 (383h7m37.980790078s) [noise=true , address=145.79.227.2:4115 , piece_id=NNHLVVAJLLRXZD7ZGLBHNWABAVZQPUU5WA65VNUZAUOLZU7SLEMA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.519080835 (383h7m37.981173318s) [noise=true , address=145.79.227.15:4874 , piece_id=RHHDISFENGWL4ODTENVZ7Q6EWTXY4EFFVBUIZ5QPH4B2NXL2U72Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.519577871 (383h7m37.981670354s) [noise=true , address=145.79.227.14:4838 , piece_id=WFDZDAW5GVLX6KZ3FPX2N6JXUCOY2SQOZZZSXKVUWPVAISWEOIRQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:54.383181912 (383h7m39.845274396s) [upload=337101]: Next returned nil segment Oct 9 12:38:54.383204786 (383h7m39.845297269s) [upload=337101]: Waiting for error group managing segments... Oct 9 12:38:54.690706964 (383h7m40.152799447s) [noise=true , address=45.91.116.11:4117 , piece_id=P33W7XEHJ4QQ6FUVVJVL7FHXKELPFRHXYZ5KLL4FTYPMPNCLTKKQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.741232556 (383h7m40.203325039s) [noise=true , address=45.91.116.7:4011 , piece_id=VLDFHTMZVVCM76YBWW5ZAMMUGLJEZ7KLVP6EKZHELGNFYGKEZZMA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.751247917 (383h7m40.2133404s) [noise=true , address=77.90.31.15:4698 , piece_id=FMCX2XNZ77M7KLAY2SFSFZ4HNYQV6VKPA3LRJXOY4F7H6AUSTNLQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.761663620 (383h7m40.223756103s) [noise=true , address=45.91.116.14:4189 , piece_id=GPJ4HMPQJHNYCDZ4A42BQ5B67OTFZ4WO3DQXRPUP734HMDIMIS6A, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.948001437 (383h7m40.41009391s) [noise=true , address=45.91.116.9:4262 , piece_id=EWIPZJTME7DCDFYK3H4IWINR34MXLVVPNPHDLCI5VO3D5IFPQHIA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.138744312 (383h7m40.600836795s) [noise=true , address=45.91.116.4:4938 , piece_id=H3IXKEMOL5BL4GWUUVI4KPPXBY567L2MGIGOL5E66UDHLNKXUJUA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.190920004 (383h7m40.653012488s) [noise=true , address=145.79.227.14:4838 , piece_id=WFDZDAW5GVLX6KZ3FPX2N6JXUCOY2SQOZZZSXKVUWPVAISWEOIRQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.207820096 (383h7m40.669912579s) [noise=true , address=145.79.227.15:4874 , piece_id=RHHDISFENGWL4ODTENVZ7Q6EWTXY4EFFVBUIZ5QPH4B2NXL2U72Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.208620800 (383h7m40.670713283s) [noise=true , address=145.79.227.13:4804 , piece_id=CJBUUUSFPR2QTBZGUK3G5GF4W7X4VAUNB3DR3LVKPKHEHUFRQ52Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.209164125 (383h7m40.671256598s) [noise=true , address=45.91.116.8:4047 , piece_id=3RRPBR6M2EDNMJLZVLT4K6JAZS3A2AUZU5ZIHFNFBJAQOOLAZUOA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.226397842 (383h7m40.688490315s) [noise=true , address=145.79.227.5:4223 , piece_id=65Q5AJCKZTLXGU7DNHQ2GSJJL2CC2KHHJS2VEWKOQKMG2JEHNJ5Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.226930757 (383h7m40.68902324s) [noise=true , address=145.79.227.6:4258 , piece_id=5MZVW7TJ4HFSYX27I5FA4H24AGKOUHAQTBHGHPGGO26BIHOQRM6A, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.278102458 (383h7m40.740194931s) [noise=true , address=145.79.227.2:4115 , piece_id=NNHLVVAJLLRXZD7ZGLBHNWABAVZQPUU5WA65VNUZAUOLZU7SLEMA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.278138768 (383h7m40.740231241s) [seg_pos={0 0}, upload=337101]: Segment reached optimal threshold of 13 pieces. Oct 9 12:38:55.278291768 (383h7m40.740384252s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Oct 9 12:38:55.278301257 (383h7m40.74039374s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278385878 (383h7m40.740478361s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 9 12:38:55.278392911 (383h7m40.740485394s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278499424 (383h7m40.740591907s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection Oct 9 12:38:55.278506477 (383h7m40.74059895s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278525063 (383h7m40.740617546s) [seg_pos={0 0}, upload=337101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:38:55.278609443 (383h7m40.740701926s) [upload=337101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:38:55.278615215 (383h7m40.740707698s) [upload=337101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:38:55.655420291 (383h7m41.117512774s) [upload=337101]: Done uploading segments... Oct 9 12:38:55.655444056 (383h7m41.117536539s) [upload=337101]: Upload finished. err: <nil> Duration: 10.471233659s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................................X...X...X..XXX......X.X] duration: 30.002869629s FAIL (2025-10-09 12:39:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 12:38:52.218563236 (383h7m37.680655719s) [upload=337101]: Starting upload Oct 9 12:38:52.218635634 (383h7m37.680728117s) [upload=337101]: Done starting upload Oct 9 12:38:52.218732439 (383h7m37.680824922s) [upload=337101]: Uploading segments... Oct 9 12:38:52.218878457 (383h7m37.68097094s) [upload=337101]: Got next segment. Inline: false Oct 9 12:38:52.218915958 (383h7m37.681008431s) [upload=337101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:38:52.218920767 (383h7m37.68101325s) [upload=337101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:38:52.512937187 (383h7m37.97502968s) [seg_pos={0 0}, upload=337101]: Begin upload segment... Oct 9 12:38:52.513008152 (383h7m37.975100635s) [seg_pos={0 0}, upload=337101]: Done begin upload segment. Oct 9 12:38:52.513042978 (383h7m37.975135461s) [noise=true , address=145.79.227.13:4804 , piece_id=CJBUUUSFPR2QTBZGUK3G5GF4W7X4VAUNB3DR3LVKPKHEHUFRQ52Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513052757 (383h7m37.97514524s) [noise=true , address=45.91.116.11:4117 , piece_id=P33W7XEHJ4QQ6FUVVJVL7FHXKELPFRHXYZ5KLL4FTYPMPNCLTKKQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513106318 (383h7m37.975198801s) [noise=true , address=45.91.116.8:4047 , piece_id=3RRPBR6M2EDNMJLZVLT4K6JAZS3A2AUZU5ZIHFNFBJAQOOLAZUOA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.513579680 (383h7m37.975672163s) [noise=true , address=145.79.227.6:4258 , piece_id=5MZVW7TJ4HFSYX27I5FA4H24AGKOUHAQTBHGHPGGO26BIHOQRM6A, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514034265 (383h7m37.976126748s) [noise=true , address=45.91.116.4:4938 , piece_id=H3IXKEMOL5BL4GWUUVI4KPPXBY567L2MGIGOL5E66UDHLNKXUJUA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514429448 (383h7m37.976521932s) [noise=true , address=45.91.116.14:4189 , piece_id=GPJ4HMPQJHNYCDZ4A42BQ5B67OTFZ4WO3DQXRPUP734HMDIMIS6A, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.514880015 (383h7m37.976972499s) [noise=true , address=145.79.227.5:4223 , piece_id=65Q5AJCKZTLXGU7DNHQ2GSJJL2CC2KHHJS2VEWKOQKMG2JEHNJ5Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.515321266 (383h7m37.977413749s) [noise=true , address=77.90.31.15:4698 , piece_id=FMCX2XNZ77M7KLAY2SFSFZ4HNYQV6VKPA3LRJXOY4F7H6AUSTNLQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.515771233 (383h7m37.977863716s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.516919299 (383h7m37.979011782s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.517471871 (383h7m37.979564354s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518002952 (383h7m37.980095435s) [noise=true , address=45.91.116.9:4262 , piece_id=EWIPZJTME7DCDFYK3H4IWINR34MXLVVPNPHDLCI5VO3D5IFPQHIA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518550345 (383h7m37.980642828s) [noise=true , address=45.91.116.7:4011 , piece_id=VLDFHTMZVVCM76YBWW5ZAMMUGLJEZ7KLVP6EKZHELGNFYGKEZZMA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.518697595 (383h7m37.980790078s) [noise=true , address=145.79.227.2:4115 , piece_id=NNHLVVAJLLRXZD7ZGLBHNWABAVZQPUU5WA65VNUZAUOLZU7SLEMA, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.519080835 (383h7m37.981173318s) [noise=true , address=145.79.227.15:4874 , piece_id=RHHDISFENGWL4ODTENVZ7Q6EWTXY4EFFVBUIZ5QPH4B2NXL2U72Q, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:52.519577871 (383h7m37.981670354s) [noise=true , address=145.79.227.14:4838 , piece_id=WFDZDAW5GVLX6KZ3FPX2N6JXUCOY2SQOZZZSXKVUWPVAISWEOIRQ, seg_pos={0 0}, upload=337101]: Uploading piece... Oct 9 12:38:54.383181912 (383h7m39.845274396s) [upload=337101]: Next returned nil segment Oct 9 12:38:54.383204786 (383h7m39.845297269s) [upload=337101]: Waiting for error group managing segments... Oct 9 12:38:54.690706964 (383h7m40.152799447s) [noise=true , address=45.91.116.11:4117 , piece_id=P33W7XEHJ4QQ6FUVVJVL7FHXKELPFRHXYZ5KLL4FTYPMPNCLTKKQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.741232556 (383h7m40.203325039s) [noise=true , address=45.91.116.7:4011 , piece_id=VLDFHTMZVVCM76YBWW5ZAMMUGLJEZ7KLVP6EKZHELGNFYGKEZZMA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.751247917 (383h7m40.2133404s) [noise=true , address=77.90.31.15:4698 , piece_id=FMCX2XNZ77M7KLAY2SFSFZ4HNYQV6VKPA3LRJXOY4F7H6AUSTNLQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.761663620 (383h7m40.223756103s) [noise=true , address=45.91.116.14:4189 , piece_id=GPJ4HMPQJHNYCDZ4A42BQ5B67OTFZ4WO3DQXRPUP734HMDIMIS6A, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:54.948001437 (383h7m40.41009391s) [noise=true , address=45.91.116.9:4262 , piece_id=EWIPZJTME7DCDFYK3H4IWINR34MXLVVPNPHDLCI5VO3D5IFPQHIA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.138744312 (383h7m40.600836795s) [noise=true , address=45.91.116.4:4938 , piece_id=H3IXKEMOL5BL4GWUUVI4KPPXBY567L2MGIGOL5E66UDHLNKXUJUA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.190920004 (383h7m40.653012488s) [noise=true , address=145.79.227.14:4838 , piece_id=WFDZDAW5GVLX6KZ3FPX2N6JXUCOY2SQOZZZSXKVUWPVAISWEOIRQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.207820096 (383h7m40.669912579s) [noise=true , address=145.79.227.15:4874 , piece_id=RHHDISFENGWL4ODTENVZ7Q6EWTXY4EFFVBUIZ5QPH4B2NXL2U72Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.208620800 (383h7m40.670713283s) [noise=true , address=145.79.227.13:4804 , piece_id=CJBUUUSFPR2QTBZGUK3G5GF4W7X4VAUNB3DR3LVKPKHEHUFRQ52Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.209164125 (383h7m40.671256598s) [noise=true , address=45.91.116.8:4047 , piece_id=3RRPBR6M2EDNMJLZVLT4K6JAZS3A2AUZU5ZIHFNFBJAQOOLAZUOA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.226397842 (383h7m40.688490315s) [noise=true , address=145.79.227.5:4223 , piece_id=65Q5AJCKZTLXGU7DNHQ2GSJJL2CC2KHHJS2VEWKOQKMG2JEHNJ5Q, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.226930757 (383h7m40.68902324s) [noise=true , address=145.79.227.6:4258 , piece_id=5MZVW7TJ4HFSYX27I5FA4H24AGKOUHAQTBHGHPGGO26BIHOQRM6A, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.278102458 (383h7m40.740194931s) [noise=true , address=145.79.227.2:4115 , piece_id=NNHLVVAJLLRXZD7ZGLBHNWABAVZQPUU5WA65VNUZAUOLZU7SLEMA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: <nil> Oct 9 12:38:55.278138768 (383h7m40.740231241s) [seg_pos={0 0}, upload=337101]: Segment reached optimal threshold of 13 pieces. Oct 9 12:38:55.278291768 (383h7m40.740384252s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Oct 9 12:38:55.278301257 (383h7m40.74039374s) [noise=true , address=145.79.227.11:4732 , piece_id=VZOPOH3VMEJEIWEZBF65ZROUZNKT4QIL5DSBY26GTMXZY4X4ZAEQ, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278385878 (383h7m40.740478361s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Oct 9 12:38:55.278392911 (383h7m40.740485394s) [noise=true , address=77.90.31.21:4879 , piece_id=DNJ7H7HHJKAHZQ6EL65O65ZEMMAQBRMT55IIEGKOA7LAV6KWGPQA, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278499424 (383h7m40.740591907s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49554->45.91.116.7:4010: use of closed network connection Oct 9 12:38:55.278506477 (383h7m40.74059895s) [noise=true , address=45.91.116.7:4010 , piece_id=WQAYYBJ3HGTUZ5FPZDIYQRCEOZQYB5CQKDJNINJBLQGZZAGTRXAQ, seg_pos={0 0}, upload=337101]: Long tail cancellation due to optimal threshold Oct 9 12:38:55.278525063 (383h7m40.740617546s) [seg_pos={0 0}, upload=337101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:38:55.278609443 (383h7m40.740701926s) [upload=337101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:38:55.278615215 (383h7m40.740707698s) [upload=337101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:38:55.655420291 (383h7m41.117512774s) [upload=337101]: Done uploading segments... Oct 9 12:38:55.655444056 (383h7m41.117536539s) [upload=337101]: Upload finished. err: <nil> Duration: 30.002895347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......................................X...X...X..XXX......X.] duration: 8.605354002s FAIL (2025-10-09 12:38:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 12:37:05.999421437 (383h5m51.46151392s) [upload=337072]: Starting upload Oct 9 12:37:05.999518823 (383h5m51.461611316s) [upload=337072]: Done starting upload Oct 9 12:37:05.999795159 (383h5m51.461887642s) [upload=337072]: Uploading segments... Oct 9 12:37:05.999935847 (383h5m51.46202834s) [upload=337072]: Got next segment. Inline: false Oct 9 12:37:06.000004868 (383h5m51.462097371s) [upload=337072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:37:06.000012843 (383h5m51.462105326s) [upload=337072]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:37:06.263423533 (383h5m51.725516016s) [seg_pos={0 0}, upload=337072]: Begin upload segment... Oct 9 12:37:06.263628132 (383h5m51.725720616s) [seg_pos={0 0}, upload=337072]: Done begin upload segment. Oct 9 12:37:06.263694979 (383h5m51.725787462s) [noise=true , address=45.91.116.8:4045 , piece_id=BT5KOCZKS3OZPQVRMO3VZ6H3EHGVXQTOJMZUY33QW7IN2MQVBC3Q, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.263727541 (383h5m51.725820024s) [noise=true , address=145.79.227.5:4222 , piece_id=HY5VSCAHXDFQJFJ7WAMB4LXYW5PMPIQLMI7GDC2LKQ77XGBWPHDQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.264330209 (383h5m51.726422692s) [noise=true , address=145.79.227.3:4151 , piece_id=NH34WRM3NGHYQ4KM4OQOHTXL2675QSL5ZIEX6YJBASNCUYVHGJMQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.264983883 (383h5m51.727076366s) [noise=true , address=77.90.31.21:4879 , piece_id=4RSEXJXU6UDB77UT5OLYFVHWVJLAWFE3KXWTDAPYEAPTWSCWFGTQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.265522770 (383h5m51.727615253s) [noise=true , address=145.79.227.7:4295 , piece_id=NP2F63RGU7J7GPCP2WCN4FMFEPUC4HO7CTZSACQBWX75ED6VBXJQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.266005008 (383h5m51.728097491s) [noise=true , address=145.79.227.18:4005 , piece_id=FUV252SVKLOSUQBTWG2MS42MJHAKVIZYAPLX37CENOQDGINTB7GA, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.266467609 (383h5m51.728560092s) [noise=true , address=45.91.116.6:4974 , piece_id=2DUUVWI3D6D5W6ZYHCSB5OSIWKN6CMQGWHU24QYR32QOBJPNG7FQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.266918207 (383h5m51.72901069s) [noise=true , address=77.90.31.18:4805 , piece_id=J7DFDQ22MJXJQYOTXOBEGMXCZXTBXSV2LNYVJEQDTMOMJKNRDB7Q, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.267792281 (383h5m51.729884764s) [noise=true , address=145.79.227.3:4150 , piece_id=H5RWXSTULCCTONNO4XBY2NWPNBMJGVMHQTZU2RDW5VXTJPG3ADKA, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.268844104 (383h5m51.730936587s) [noise=true , address=145.79.227.6:4258 , piece_id=ZE7TEZJI2LOQIFZMZ2WXXXKBMQDAVHKKEVKQ242OJ5UP4C6KINUQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.268951909 (383h5m51.731044392s) [noise=true , address=145.79.227.15:4876 , piece_id=CIBKHZV4VGIRWG6RYVX22YZT4EUCISQTOHZBQCUKM2Q74SDN4MEQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.270202901 (383h5m51.732295374s) [noise=true , address=145.79.227.13:4801 , piece_id=MLCVL7EVFGTJRYODAHQ5XAZOVECYHEVBPNEVGB33MRAMLSHOXTMA, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.271900955 (383h5m51.733993438s) [noise=true , address=145.79.227.11:4730 , piece_id=GXOSAJQZAZGFWZM33IIYX33T24DIG4WIYUKE6ELBDCDTHRJPKZJA, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.273535026 (383h5m51.735627499s) [noise=true , address=77.90.31.14:4661 , piece_id=72P6ZWYT2KIWI6QDMSCK2DNMHCBW3KU2RNSK4ADKTBNS3NQTVOMA, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.275114955 (383h5m51.737207438s) [noise=true , address=77.90.31.12:4592 , piece_id=JVJURHVEDLARWCNTAVHDMIGDJBEIDQB75OJY7Z4Y2TST64UUG6HQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.277111487 (383h5m51.73920397s) [noise=true , address=145.79.227.14:4837 , piece_id=QP3E24ADAWCCS2AYRWR5SE4EGBSKDM2WWBGFWA6IPPWXYQFK5ERA, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:08.162309768 (383h5m53.624402251s) [upload=337072]: Next returned nil segment Oct 9 12:37:08.162334856 (383h5m53.624427339s) [upload=337072]: Waiting for error group managing segments... Oct 9 12:37:08.469698280 (383h5m53.931790764s) [noise=true , address=77.90.31.14:4661 , piece_id=72P6ZWYT2KIWI6QDMSCK2DNMHCBW3KU2RNSK4ADKTBNS3NQTVOMA, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:08.579635925 (383h5m54.041728398s) [noise=true , address=77.90.31.12:4592 , piece_id=JVJURHVEDLARWCNTAVHDMIGDJBEIDQB75OJY7Z4Y2TST64UUG6HQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:08.761766499 (383h5m54.223858992s) [noise=true , address=77.90.31.21:4879 , piece_id=4RSEXJXU6UDB77UT5OLYFVHWVJLAWFE3KXWTDAPYEAPTWSCWFGTQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:08.762572403 (383h5m54.224664886s) [noise=true , address=45.91.116.8:4045 , piece_id=BT5KOCZKS3OZPQVRMO3VZ6H3EHGVXQTOJMZUY33QW7IN2MQVBC3Q, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:08.866972202 (383h5m54.329064685s) [noise=true , address=45.91.116.6:4974 , piece_id=2DUUVWI3D6D5W6ZYHCSB5OSIWKN6CMQGWHU24QYR32QOBJPNG7FQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.006724941 (383h5m54.468817424s) [noise=true , address=145.79.227.3:4151 , piece_id=NH34WRM3NGHYQ4KM4OQOHTXL2675QSL5ZIEX6YJBASNCUYVHGJMQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.024164119 (383h5m54.486256602s) [noise=true , address=145.79.227.15:4876 , piece_id=CIBKHZV4VGIRWG6RYVX22YZT4EUCISQTOHZBQCUKM2Q74SDN4MEQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.049956914 (383h5m54.512049397s) [noise=true , address=145.79.227.5:4222 , piece_id=HY5VSCAHXDFQJFJ7WAMB4LXYW5PMPIQLMI7GDC2LKQ77XGBWPHDQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.060710110 (383h5m54.522802593s) [noise=true , address=145.79.227.7:4295 , piece_id=NP2F63RGU7J7GPCP2WCN4FMFEPUC4HO7CTZSACQBWX75ED6VBXJQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.060764734 (383h5m54.522857217s) [noise=true , address=145.79.227.6:4258 , piece_id=ZE7TEZJI2LOQIFZMZ2WXXXKBMQDAVHKKEVKQ242OJ5UP4C6KINUQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.064683787 (383h5m54.52677626s) [noise=true , address=145.79.227.3:4150 , piece_id=H5RWXSTULCCTONNO4XBY2NWPNBMJGVMHQTZU2RDW5VXTJPG3ADKA, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.088696591 (383h5m54.550789074s) [noise=true , address=145.79.227.13:4801 , piece_id=MLCVL7EVFGTJRYODAHQ5XAZOVECYHEVBPNEVGB33MRAMLSHOXTMA, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.103393117 (383h5m54.56548559s) [noise=true , address=145.79.227.14:4837 , piece_id=QP3E24ADAWCCS2AYRWR5SE4EGBSKDM2WWBGFWA6IPPWXYQFK5ERA, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.103417043 (383h5m54.565509526s) [seg_pos={0 0}, upload=337072]: Segment reached optimal threshold of 13 pieces. Oct 9 12:37:09.103544776 (383h5m54.565637259s) [noise=true , address=145.79.227.18:4005 , piece_id=FUV252SVKLOSUQBTWG2MS42MJHAKVIZYAPLX37CENOQDGINTB7GA, seg_pos={0 0}, upload=337072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: expected piece hash; context canceled; EOF Oct 9 12:37:09.103555637 (383h5m54.56564811s) [noise=true , address=145.79.227.18:4005 , piece_id=FUV252SVKLOSUQBTWG2MS42MJHAKVIZYAPLX37CENOQDGINTB7GA, seg_pos={0 0}, upload=337072]: Long tail cancellation due to optimal threshold Oct 9 12:37:09.103674913 (383h5m54.565767386s) [noise=true , address=145.79.227.11:4730 , piece_id=GXOSAJQZAZGFWZM33IIYX33T24DIG4WIYUKE6ELBDCDTHRJPKZJA, seg_pos={0 0}, upload=337072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Oct 9 12:37:09.103683019 (383h5m54.565775503s) [noise=true , address=145.79.227.11:4730 , piece_id=GXOSAJQZAZGFWZM33IIYX33T24DIG4WIYUKE6ELBDCDTHRJPKZJA, seg_pos={0 0}, upload=337072]: Long tail cancellation due to optimal threshold Oct 9 12:37:09.103803749 (383h5m54.565896232s) [noise=true , address=77.90.31.18:4805 , piece_id=J7DFDQ22MJXJQYOTXOBEGMXCZXTBXSV2LNYVJEQDTMOMJKNRDB7Q, seg_pos={0 0}, upload=337072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Oct 9 12:37:09.103813848 (383h5m54.565906331s) [noise=true , address=77.90.31.18:4805 , piece_id=J7DFDQ22MJXJQYOTXOBEGMXCZXTBXSV2LNYVJEQDTMOMJKNRDB7Q, seg_pos={0 0}, upload=337072]: Long tail cancellation due to optimal threshold Oct 9 12:37:09.103832704 (383h5m54.565925187s) [seg_pos={0 0}, upload=337072]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:37:09.103921553 (383h5m54.566014026s) [upload=337072]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:37:09.103929499 (383h5m54.566021982s) [upload=337072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:37:09.575522420 (383h5m55.037614903s) [upload=337072]: Done uploading segments... Oct 9 12:37:09.575545825 (383h5m55.037638308s) [upload=337072]: Upload finished. err: <nil> Duration: 8.605391114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......................................X...X...X..XXX......X] duration: 30.010002515s FAIL (2025-10-09 12:37:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 12:37:05.999421437 (383h5m51.46151392s) [upload=337072]: Starting upload Oct 9 12:37:05.999518823 (383h5m51.461611316s) [upload=337072]: Done starting upload Oct 9 12:37:05.999795159 (383h5m51.461887642s) [upload=337072]: Uploading segments... Oct 9 12:37:05.999935847 (383h5m51.46202834s) [upload=337072]: Got next segment. Inline: false Oct 9 12:37:06.000004868 (383h5m51.462097371s) [upload=337072]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:37:06.000012843 (383h5m51.462105326s) [upload=337072]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:37:06.263423533 (383h5m51.725516016s) [seg_pos={0 0}, upload=337072]: Begin upload segment... Oct 9 12:37:06.263628132 (383h5m51.725720616s) [seg_pos={0 0}, upload=337072]: Done begin upload segment. Oct 9 12:37:06.263694979 (383h5m51.725787462s) [noise=true , address=45.91.116.8:4045 , piece_id=BT5KOCZKS3OZPQVRMO3VZ6H3EHGVXQTOJMZUY33QW7IN2MQVBC3Q, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.263727541 (383h5m51.725820024s) [noise=true , address=145.79.227.5:4222 , piece_id=HY5VSCAHXDFQJFJ7WAMB4LXYW5PMPIQLMI7GDC2LKQ77XGBWPHDQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.264330209 (383h5m51.726422692s) [noise=true , address=145.79.227.3:4151 , piece_id=NH34WRM3NGHYQ4KM4OQOHTXL2675QSL5ZIEX6YJBASNCUYVHGJMQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.264983883 (383h5m51.727076366s) [noise=true , address=77.90.31.21:4879 , piece_id=4RSEXJXU6UDB77UT5OLYFVHWVJLAWFE3KXWTDAPYEAPTWSCWFGTQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.265522770 (383h5m51.727615253s) [noise=true , address=145.79.227.7:4295 , piece_id=NP2F63RGU7J7GPCP2WCN4FMFEPUC4HO7CTZSACQBWX75ED6VBXJQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.266005008 (383h5m51.728097491s) [noise=true , address=145.79.227.18:4005 , piece_id=FUV252SVKLOSUQBTWG2MS42MJHAKVIZYAPLX37CENOQDGINTB7GA, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.266467609 (383h5m51.728560092s) [noise=true , address=45.91.116.6:4974 , piece_id=2DUUVWI3D6D5W6ZYHCSB5OSIWKN6CMQGWHU24QYR32QOBJPNG7FQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.266918207 (383h5m51.72901069s) [noise=true , address=77.90.31.18:4805 , piece_id=J7DFDQ22MJXJQYOTXOBEGMXCZXTBXSV2LNYVJEQDTMOMJKNRDB7Q, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.267792281 (383h5m51.729884764s) [noise=true , address=145.79.227.3:4150 , piece_id=H5RWXSTULCCTONNO4XBY2NWPNBMJGVMHQTZU2RDW5VXTJPG3ADKA, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.268844104 (383h5m51.730936587s) [noise=true , address=145.79.227.6:4258 , piece_id=ZE7TEZJI2LOQIFZMZ2WXXXKBMQDAVHKKEVKQ242OJ5UP4C6KINUQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.268951909 (383h5m51.731044392s) [noise=true , address=145.79.227.15:4876 , piece_id=CIBKHZV4VGIRWG6RYVX22YZT4EUCISQTOHZBQCUKM2Q74SDN4MEQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.270202901 (383h5m51.732295374s) [noise=true , address=145.79.227.13:4801 , piece_id=MLCVL7EVFGTJRYODAHQ5XAZOVECYHEVBPNEVGB33MRAMLSHOXTMA, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.271900955 (383h5m51.733993438s) [noise=true , address=145.79.227.11:4730 , piece_id=GXOSAJQZAZGFWZM33IIYX33T24DIG4WIYUKE6ELBDCDTHRJPKZJA, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.273535026 (383h5m51.735627499s) [noise=true , address=77.90.31.14:4661 , piece_id=72P6ZWYT2KIWI6QDMSCK2DNMHCBW3KU2RNSK4ADKTBNS3NQTVOMA, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.275114955 (383h5m51.737207438s) [noise=true , address=77.90.31.12:4592 , piece_id=JVJURHVEDLARWCNTAVHDMIGDJBEIDQB75OJY7Z4Y2TST64UUG6HQ, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:06.277111487 (383h5m51.73920397s) [noise=true , address=145.79.227.14:4837 , piece_id=QP3E24ADAWCCS2AYRWR5SE4EGBSKDM2WWBGFWA6IPPWXYQFK5ERA, seg_pos={0 0}, upload=337072]: Uploading piece... Oct 9 12:37:08.162309768 (383h5m53.624402251s) [upload=337072]: Next returned nil segment Oct 9 12:37:08.162334856 (383h5m53.624427339s) [upload=337072]: Waiting for error group managing segments... Oct 9 12:37:08.469698280 (383h5m53.931790764s) [noise=true , address=77.90.31.14:4661 , piece_id=72P6ZWYT2KIWI6QDMSCK2DNMHCBW3KU2RNSK4ADKTBNS3NQTVOMA, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:08.579635925 (383h5m54.041728398s) [noise=true , address=77.90.31.12:4592 , piece_id=JVJURHVEDLARWCNTAVHDMIGDJBEIDQB75OJY7Z4Y2TST64UUG6HQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:08.761766499 (383h5m54.223858992s) [noise=true , address=77.90.31.21:4879 , piece_id=4RSEXJXU6UDB77UT5OLYFVHWVJLAWFE3KXWTDAPYEAPTWSCWFGTQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:08.762572403 (383h5m54.224664886s) [noise=true , address=45.91.116.8:4045 , piece_id=BT5KOCZKS3OZPQVRMO3VZ6H3EHGVXQTOJMZUY33QW7IN2MQVBC3Q, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:08.866972202 (383h5m54.329064685s) [noise=true , address=45.91.116.6:4974 , piece_id=2DUUVWI3D6D5W6ZYHCSB5OSIWKN6CMQGWHU24QYR32QOBJPNG7FQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.006724941 (383h5m54.468817424s) [noise=true , address=145.79.227.3:4151 , piece_id=NH34WRM3NGHYQ4KM4OQOHTXL2675QSL5ZIEX6YJBASNCUYVHGJMQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.024164119 (383h5m54.486256602s) [noise=true , address=145.79.227.15:4876 , piece_id=CIBKHZV4VGIRWG6RYVX22YZT4EUCISQTOHZBQCUKM2Q74SDN4MEQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.049956914 (383h5m54.512049397s) [noise=true , address=145.79.227.5:4222 , piece_id=HY5VSCAHXDFQJFJ7WAMB4LXYW5PMPIQLMI7GDC2LKQ77XGBWPHDQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.060710110 (383h5m54.522802593s) [noise=true , address=145.79.227.7:4295 , piece_id=NP2F63RGU7J7GPCP2WCN4FMFEPUC4HO7CTZSACQBWX75ED6VBXJQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.060764734 (383h5m54.522857217s) [noise=true , address=145.79.227.6:4258 , piece_id=ZE7TEZJI2LOQIFZMZ2WXXXKBMQDAVHKKEVKQ242OJ5UP4C6KINUQ, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.064683787 (383h5m54.52677626s) [noise=true , address=145.79.227.3:4150 , piece_id=H5RWXSTULCCTONNO4XBY2NWPNBMJGVMHQTZU2RDW5VXTJPG3ADKA, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.088696591 (383h5m54.550789074s) [noise=true , address=145.79.227.13:4801 , piece_id=MLCVL7EVFGTJRYODAHQ5XAZOVECYHEVBPNEVGB33MRAMLSHOXTMA, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.103393117 (383h5m54.56548559s) [noise=true , address=145.79.227.14:4837 , piece_id=QP3E24ADAWCCS2AYRWR5SE4EGBSKDM2WWBGFWA6IPPWXYQFK5ERA, seg_pos={0 0}, upload=337072]: Done uploading piece. err: <nil> Oct 9 12:37:09.103417043 (383h5m54.565509526s) [seg_pos={0 0}, upload=337072]: Segment reached optimal threshold of 13 pieces. Oct 9 12:37:09.103544776 (383h5m54.565637259s) [noise=true , address=145.79.227.18:4005 , piece_id=FUV252SVKLOSUQBTWG2MS42MJHAKVIZYAPLX37CENOQDGINTB7GA, seg_pos={0 0}, upload=337072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ): protocol: expected piece hash; context canceled; EOF Oct 9 12:37:09.103555637 (383h5m54.56564811s) [noise=true , address=145.79.227.18:4005 , piece_id=FUV252SVKLOSUQBTWG2MS42MJHAKVIZYAPLX37CENOQDGINTB7GA, seg_pos={0 0}, upload=337072]: Long tail cancellation due to optimal threshold Oct 9 12:37:09.103674913 (383h5m54.565767386s) [noise=true , address=145.79.227.11:4730 , piece_id=GXOSAJQZAZGFWZM33IIYX33T24DIG4WIYUKE6ELBDCDTHRJPKZJA, seg_pos={0 0}, upload=337072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Oct 9 12:37:09.103683019 (383h5m54.565775503s) [noise=true , address=145.79.227.11:4730 , piece_id=GXOSAJQZAZGFWZM33IIYX33T24DIG4WIYUKE6ELBDCDTHRJPKZJA, seg_pos={0 0}, upload=337072]: Long tail cancellation due to optimal threshold Oct 9 12:37:09.103803749 (383h5m54.565896232s) [noise=true , address=77.90.31.18:4805 , piece_id=J7DFDQ22MJXJQYOTXOBEGMXCZXTBXSV2LNYVJEQDTMOMJKNRDB7Q, seg_pos={0 0}, upload=337072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Oct 9 12:37:09.103813848 (383h5m54.565906331s) [noise=true , address=77.90.31.18:4805 , piece_id=J7DFDQ22MJXJQYOTXOBEGMXCZXTBXSV2LNYVJEQDTMOMJKNRDB7Q, seg_pos={0 0}, upload=337072]: Long tail cancellation due to optimal threshold Oct 9 12:37:09.103832704 (383h5m54.565925187s) [seg_pos={0 0}, upload=337072]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:37:09.103921553 (383h5m54.566014026s) [upload=337072]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:37:09.103929499 (383h5m54.566021982s) [upload=337072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:37:09.575522420 (383h5m55.037614903s) [upload=337072]: Done uploading segments... Oct 9 12:37:09.575545825 (383h5m55.037638308s) [upload=337072]: Upload finished. err: <nil> Duration: 30.010024877s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................................X...X...X..XXX......] duration: 7.37167544s FAIL (2025-10-09 12:36:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 12:32:11.014667976 (383h0m56.476760459s) [upload=336995]: Starting upload Oct 9 12:32:11.014753328 (383h0m56.476845811s) [upload=336995]: Done starting upload Oct 9 12:32:11.015033873 (383h0m56.477126366s) [upload=336995]: Uploading segments... Oct 9 12:32:11.015082205 (383h0m56.477174688s) [upload=336995]: Got next segment. Inline: false Oct 9 12:32:11.015122642 (383h0m56.477215125s) [upload=336995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:32:11.015128453 (383h0m56.477220926s) [upload=336995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:32:11.352699541 (383h0m56.814792014s) [seg_pos={0 0}, upload=336995]: Begin upload segment... Oct 9 12:32:11.352754256 (383h0m56.814846739s) [seg_pos={0 0}, upload=336995]: Done begin upload segment. Oct 9 12:32:11.352784163 (383h0m56.814876646s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.353342336 (383h0m56.815434819s) [noise=true , address=45.91.116.11:4118 , piece_id=GW7B7NELLNAI42Z2CJMGGJHATPQHYTWDJALC3TNRX365456B4VLQ, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354029434 (383h0m56.816121927s) [noise=true , address=145.79.227.5:4222 , piece_id=QPCB2VXUWG22EUQ6WJTEYY42ZDCUGTO6K3OWPQWBOJZJMDBPEWFA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354447490 (383h0m56.816539973s) [noise=true , address=45.91.116.5:4298 , piece_id=OCGLBI4L424PQ2PFWS4WYMQCVMWPRWDN3IZY5PAB7XV2ESQ52BDA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354877619 (383h0m56.816970112s) [noise=true , address=145.79.227.13:4803 , piece_id=2BHC3PWTD65IGVGN4653DX36X6RLHXDM5CJSLMJ6L7ZSLREJWT2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.355353105 (383h0m56.817445588s) [noise=true , address=145.79.227.2:4115 , piece_id=XVKTC2ZKQJL6IREHVUIROSOHMD6YUPAHSQDRSJHP4YIAFJNPWCGA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.355789164 (383h0m56.817881647s) [noise=true , address=77.90.31.17:4772 , piece_id=S5PQWDPOBY7TTFAIYZ3YTYBYVH5CBI5CN5D44ZW2HHUHXIWNGKNA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.356805349 (383h0m56.818897832s) [noise=true , address=45.91.116.9:4262 , piece_id=LDX4BO2OK6OKZMJZJBDXTVKECZUE22BYIHZII5FL7TGPFKAWDZTA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.357222053 (383h0m56.819314536s) [noise=true , address=77.90.31.15:4697 , piece_id=KEIBEPNKBMQSIVUTUQYTG6RTYRR5CLF2I5PHSTPN7XIWJ7ME26XA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.358482193 (383h0m56.820574676s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.359022401 (383h0m56.821114894s) [noise=true , address=145.79.227.3:4150 , piece_id=IAC57SI5LKOJVARZHLCVFIJBK4Q5RYFWP3TR3Z2MFSTUPVSGV53Q, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.359922265 (383h0m56.822014748s) [noise=true , address=45.91.116.6:4975 , piece_id=57NQURKND6OPNKNHSKSC4BII5TR2X2WVYJV2UAVN5M7KCSTAHY3A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.360618491 (383h0m56.822710974s) [noise=true , address=45.91.116.7:4009 , piece_id=M7YASOBZYEOLOTHRC3KQ6NOVZZHQENHBM5KWLF54M53CHVXVDB2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.361398246 (383h0m56.823490729s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.362845992 (383h0m56.824938465s) [noise=true , address=145.79.227.11:4730 , piece_id=4BMR36W56SM2QWQQWD3LRUPZ4GI22NBVOJPEQD5WVRL4LD66GVIA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.364180013 (383h0m56.826272496s) [noise=true , address=145.79.227.14:4838 , piece_id=S7IRMYZAGLSDRCMMHAGCJ6476AHFMWW4ZKW74IB2G22XWBQVUTSA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:13.299270278 (383h0m58.761362761s) [upload=336995]: Next returned nil segment Oct 9 12:32:13.299290196 (383h0m58.761382679s) [upload=336995]: Waiting for error group managing segments... Oct 9 12:32:13.615010146 (383h0m59.077102629s) [noise=true , address=45.91.116.5:4298 , piece_id=OCGLBI4L424PQ2PFWS4WYMQCVMWPRWDN3IZY5PAB7XV2ESQ52BDA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.646135705 (383h0m59.108228188s) [noise=true , address=45.91.116.11:4118 , piece_id=GW7B7NELLNAI42Z2CJMGGJHATPQHYTWDJALC3TNRX365456B4VLQ, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.725132132 (383h0m59.187224625s) [noise=true , address=77.90.31.15:4697 , piece_id=KEIBEPNKBMQSIVUTUQYTG6RTYRR5CLF2I5PHSTPN7XIWJ7ME26XA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.831029063 (383h0m59.293121546s) [noise=true , address=45.91.116.6:4975 , piece_id=57NQURKND6OPNKNHSKSC4BII5TR2X2WVYJV2UAVN5M7KCSTAHY3A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.844060108 (383h0m59.306152591s) [noise=true , address=77.90.31.17:4772 , piece_id=S5PQWDPOBY7TTFAIYZ3YTYBYVH5CBI5CN5D44ZW2HHUHXIWNGKNA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.924210421 (383h0m59.386302894s) [noise=true , address=45.91.116.7:4009 , piece_id=M7YASOBZYEOLOTHRC3KQ6NOVZZHQENHBM5KWLF54M53CHVXVDB2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.115471763 (383h0m59.577564247s) [noise=true , address=45.91.116.9:4262 , piece_id=LDX4BO2OK6OKZMJZJBDXTVKECZUE22BYIHZII5FL7TGPFKAWDZTA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.137527292 (383h0m59.599619776s) [noise=true , address=145.79.227.13:4803 , piece_id=2BHC3PWTD65IGVGN4653DX36X6RLHXDM5CJSLMJ6L7ZSLREJWT2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.139239683 (383h0m59.601332166s) [noise=true , address=145.79.227.14:4838 , piece_id=S7IRMYZAGLSDRCMMHAGCJ6476AHFMWW4ZKW74IB2G22XWBQVUTSA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.149064161 (383h0m59.611156644s) [noise=true , address=145.79.227.11:4730 , piece_id=4BMR36W56SM2QWQQWD3LRUPZ4GI22NBVOJPEQD5WVRL4LD66GVIA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.156402325 (383h0m59.618494808s) [noise=true , address=145.79.227.5:4222 , piece_id=QPCB2VXUWG22EUQ6WJTEYY42ZDCUGTO6K3OWPQWBOJZJMDBPEWFA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.178407738 (383h0m59.640500241s) [noise=true , address=145.79.227.2:4115 , piece_id=XVKTC2ZKQJL6IREHVUIROSOHMD6YUPAHSQDRSJHP4YIAFJNPWCGA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.246646118 (383h0m59.708738601s) [noise=true , address=145.79.227.3:4150 , piece_id=IAC57SI5LKOJVARZHLCVFIJBK4Q5RYFWP3TR3Z2MFSTUPVSGV53Q, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.246687597 (383h0m59.70878008s) [seg_pos={0 0}, upload=336995]: Segment reached optimal threshold of 13 pieces. Oct 9 12:32:14.246874583 (383h0m59.708967066s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246885854 (383h0m59.708978338s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246910331 (383h0m59.709002824s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246923005 (383h0m59.709015488s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246965816 (383h0m59.7090583s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246975606 (383h0m59.709068089s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246996415 (383h0m59.709088898s) [seg_pos={0 0}, upload=336995]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:32:14.247095574 (383h0m59.709188067s) [upload=336995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:32:14.247103950 (383h0m59.709196433s) [upload=336995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:32:14.670708184 (383h1m0.132800667s) [upload=336995]: Done uploading segments... Oct 9 12:32:14.670733413 (383h1m0.132825896s) [upload=336995]: Upload finished. err: <nil> Duration: 7.371709365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................................X...X...X..XXX.....] duration: 8.517642995s FAIL (2025-10-09 12:36:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 12:32:11.014667976 (383h0m56.476760459s) [upload=336995]: Starting upload Oct 9 12:32:11.014753328 (383h0m56.476845811s) [upload=336995]: Done starting upload Oct 9 12:32:11.015033873 (383h0m56.477126366s) [upload=336995]: Uploading segments... Oct 9 12:32:11.015082205 (383h0m56.477174688s) [upload=336995]: Got next segment. Inline: false Oct 9 12:32:11.015122642 (383h0m56.477215125s) [upload=336995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:32:11.015128453 (383h0m56.477220926s) [upload=336995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:32:11.352699541 (383h0m56.814792014s) [seg_pos={0 0}, upload=336995]: Begin upload segment... Oct 9 12:32:11.352754256 (383h0m56.814846739s) [seg_pos={0 0}, upload=336995]: Done begin upload segment. Oct 9 12:32:11.352784163 (383h0m56.814876646s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.353342336 (383h0m56.815434819s) [noise=true , address=45.91.116.11:4118 , piece_id=GW7B7NELLNAI42Z2CJMGGJHATPQHYTWDJALC3TNRX365456B4VLQ, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354029434 (383h0m56.816121927s) [noise=true , address=145.79.227.5:4222 , piece_id=QPCB2VXUWG22EUQ6WJTEYY42ZDCUGTO6K3OWPQWBOJZJMDBPEWFA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354447490 (383h0m56.816539973s) [noise=true , address=45.91.116.5:4298 , piece_id=OCGLBI4L424PQ2PFWS4WYMQCVMWPRWDN3IZY5PAB7XV2ESQ52BDA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354877619 (383h0m56.816970112s) [noise=true , address=145.79.227.13:4803 , piece_id=2BHC3PWTD65IGVGN4653DX36X6RLHXDM5CJSLMJ6L7ZSLREJWT2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.355353105 (383h0m56.817445588s) [noise=true , address=145.79.227.2:4115 , piece_id=XVKTC2ZKQJL6IREHVUIROSOHMD6YUPAHSQDRSJHP4YIAFJNPWCGA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.355789164 (383h0m56.817881647s) [noise=true , address=77.90.31.17:4772 , piece_id=S5PQWDPOBY7TTFAIYZ3YTYBYVH5CBI5CN5D44ZW2HHUHXIWNGKNA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.356805349 (383h0m56.818897832s) [noise=true , address=45.91.116.9:4262 , piece_id=LDX4BO2OK6OKZMJZJBDXTVKECZUE22BYIHZII5FL7TGPFKAWDZTA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.357222053 (383h0m56.819314536s) [noise=true , address=77.90.31.15:4697 , piece_id=KEIBEPNKBMQSIVUTUQYTG6RTYRR5CLF2I5PHSTPN7XIWJ7ME26XA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.358482193 (383h0m56.820574676s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.359022401 (383h0m56.821114894s) [noise=true , address=145.79.227.3:4150 , piece_id=IAC57SI5LKOJVARZHLCVFIJBK4Q5RYFWP3TR3Z2MFSTUPVSGV53Q, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.359922265 (383h0m56.822014748s) [noise=true , address=45.91.116.6:4975 , piece_id=57NQURKND6OPNKNHSKSC4BII5TR2X2WVYJV2UAVN5M7KCSTAHY3A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.360618491 (383h0m56.822710974s) [noise=true , address=45.91.116.7:4009 , piece_id=M7YASOBZYEOLOTHRC3KQ6NOVZZHQENHBM5KWLF54M53CHVXVDB2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.361398246 (383h0m56.823490729s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.362845992 (383h0m56.824938465s) [noise=true , address=145.79.227.11:4730 , piece_id=4BMR36W56SM2QWQQWD3LRUPZ4GI22NBVOJPEQD5WVRL4LD66GVIA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.364180013 (383h0m56.826272496s) [noise=true , address=145.79.227.14:4838 , piece_id=S7IRMYZAGLSDRCMMHAGCJ6476AHFMWW4ZKW74IB2G22XWBQVUTSA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:13.299270278 (383h0m58.761362761s) [upload=336995]: Next returned nil segment Oct 9 12:32:13.299290196 (383h0m58.761382679s) [upload=336995]: Waiting for error group managing segments... Oct 9 12:32:13.615010146 (383h0m59.077102629s) [noise=true , address=45.91.116.5:4298 , piece_id=OCGLBI4L424PQ2PFWS4WYMQCVMWPRWDN3IZY5PAB7XV2ESQ52BDA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.646135705 (383h0m59.108228188s) [noise=true , address=45.91.116.11:4118 , piece_id=GW7B7NELLNAI42Z2CJMGGJHATPQHYTWDJALC3TNRX365456B4VLQ, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.725132132 (383h0m59.187224625s) [noise=true , address=77.90.31.15:4697 , piece_id=KEIBEPNKBMQSIVUTUQYTG6RTYRR5CLF2I5PHSTPN7XIWJ7ME26XA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.831029063 (383h0m59.293121546s) [noise=true , address=45.91.116.6:4975 , piece_id=57NQURKND6OPNKNHSKSC4BII5TR2X2WVYJV2UAVN5M7KCSTAHY3A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.844060108 (383h0m59.306152591s) [noise=true , address=77.90.31.17:4772 , piece_id=S5PQWDPOBY7TTFAIYZ3YTYBYVH5CBI5CN5D44ZW2HHUHXIWNGKNA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.924210421 (383h0m59.386302894s) [noise=true , address=45.91.116.7:4009 , piece_id=M7YASOBZYEOLOTHRC3KQ6NOVZZHQENHBM5KWLF54M53CHVXVDB2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.115471763 (383h0m59.577564247s) [noise=true , address=45.91.116.9:4262 , piece_id=LDX4BO2OK6OKZMJZJBDXTVKECZUE22BYIHZII5FL7TGPFKAWDZTA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.137527292 (383h0m59.599619776s) [noise=true , address=145.79.227.13:4803 , piece_id=2BHC3PWTD65IGVGN4653DX36X6RLHXDM5CJSLMJ6L7ZSLREJWT2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.139239683 (383h0m59.601332166s) [noise=true , address=145.79.227.14:4838 , piece_id=S7IRMYZAGLSDRCMMHAGCJ6476AHFMWW4ZKW74IB2G22XWBQVUTSA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.149064161 (383h0m59.611156644s) [noise=true , address=145.79.227.11:4730 , piece_id=4BMR36W56SM2QWQQWD3LRUPZ4GI22NBVOJPEQD5WVRL4LD66GVIA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.156402325 (383h0m59.618494808s) [noise=true , address=145.79.227.5:4222 , piece_id=QPCB2VXUWG22EUQ6WJTEYY42ZDCUGTO6K3OWPQWBOJZJMDBPEWFA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.178407738 (383h0m59.640500241s) [noise=true , address=145.79.227.2:4115 , piece_id=XVKTC2ZKQJL6IREHVUIROSOHMD6YUPAHSQDRSJHP4YIAFJNPWCGA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.246646118 (383h0m59.708738601s) [noise=true , address=145.79.227.3:4150 , piece_id=IAC57SI5LKOJVARZHLCVFIJBK4Q5RYFWP3TR3Z2MFSTUPVSGV53Q, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.246687597 (383h0m59.70878008s) [seg_pos={0 0}, upload=336995]: Segment reached optimal threshold of 13 pieces. Oct 9 12:32:14.246874583 (383h0m59.708967066s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246885854 (383h0m59.708978338s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246910331 (383h0m59.709002824s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246923005 (383h0m59.709015488s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246965816 (383h0m59.7090583s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246975606 (383h0m59.709068089s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246996415 (383h0m59.709088898s) [seg_pos={0 0}, upload=336995]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:32:14.247095574 (383h0m59.709188067s) [upload=336995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:32:14.247103950 (383h0m59.709196433s) [upload=336995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:32:14.670708184 (383h1m0.132800667s) [upload=336995]: Done uploading segments... Oct 9 12:32:14.670733413 (383h1m0.132825896s) [upload=336995]: Upload finished. err: <nil> Duration: 8.517679505s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................................X...X...X..XXX....] duration: 7.579792173s FAIL (2025-10-09 12:35:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 12:32:11.014667976 (383h0m56.476760459s) [upload=336995]: Starting upload Oct 9 12:32:11.014753328 (383h0m56.476845811s) [upload=336995]: Done starting upload Oct 9 12:32:11.015033873 (383h0m56.477126366s) [upload=336995]: Uploading segments... Oct 9 12:32:11.015082205 (383h0m56.477174688s) [upload=336995]: Got next segment. Inline: false Oct 9 12:32:11.015122642 (383h0m56.477215125s) [upload=336995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:32:11.015128453 (383h0m56.477220926s) [upload=336995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:32:11.352699541 (383h0m56.814792014s) [seg_pos={0 0}, upload=336995]: Begin upload segment... Oct 9 12:32:11.352754256 (383h0m56.814846739s) [seg_pos={0 0}, upload=336995]: Done begin upload segment. Oct 9 12:32:11.352784163 (383h0m56.814876646s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.353342336 (383h0m56.815434819s) [noise=true , address=45.91.116.11:4118 , piece_id=GW7B7NELLNAI42Z2CJMGGJHATPQHYTWDJALC3TNRX365456B4VLQ, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354029434 (383h0m56.816121927s) [noise=true , address=145.79.227.5:4222 , piece_id=QPCB2VXUWG22EUQ6WJTEYY42ZDCUGTO6K3OWPQWBOJZJMDBPEWFA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354447490 (383h0m56.816539973s) [noise=true , address=45.91.116.5:4298 , piece_id=OCGLBI4L424PQ2PFWS4WYMQCVMWPRWDN3IZY5PAB7XV2ESQ52BDA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354877619 (383h0m56.816970112s) [noise=true , address=145.79.227.13:4803 , piece_id=2BHC3PWTD65IGVGN4653DX36X6RLHXDM5CJSLMJ6L7ZSLREJWT2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.355353105 (383h0m56.817445588s) [noise=true , address=145.79.227.2:4115 , piece_id=XVKTC2ZKQJL6IREHVUIROSOHMD6YUPAHSQDRSJHP4YIAFJNPWCGA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.355789164 (383h0m56.817881647s) [noise=true , address=77.90.31.17:4772 , piece_id=S5PQWDPOBY7TTFAIYZ3YTYBYVH5CBI5CN5D44ZW2HHUHXIWNGKNA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.356805349 (383h0m56.818897832s) [noise=true , address=45.91.116.9:4262 , piece_id=LDX4BO2OK6OKZMJZJBDXTVKECZUE22BYIHZII5FL7TGPFKAWDZTA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.357222053 (383h0m56.819314536s) [noise=true , address=77.90.31.15:4697 , piece_id=KEIBEPNKBMQSIVUTUQYTG6RTYRR5CLF2I5PHSTPN7XIWJ7ME26XA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.358482193 (383h0m56.820574676s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.359022401 (383h0m56.821114894s) [noise=true , address=145.79.227.3:4150 , piece_id=IAC57SI5LKOJVARZHLCVFIJBK4Q5RYFWP3TR3Z2MFSTUPVSGV53Q, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.359922265 (383h0m56.822014748s) [noise=true , address=45.91.116.6:4975 , piece_id=57NQURKND6OPNKNHSKSC4BII5TR2X2WVYJV2UAVN5M7KCSTAHY3A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.360618491 (383h0m56.822710974s) [noise=true , address=45.91.116.7:4009 , piece_id=M7YASOBZYEOLOTHRC3KQ6NOVZZHQENHBM5KWLF54M53CHVXVDB2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.361398246 (383h0m56.823490729s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.362845992 (383h0m56.824938465s) [noise=true , address=145.79.227.11:4730 , piece_id=4BMR36W56SM2QWQQWD3LRUPZ4GI22NBVOJPEQD5WVRL4LD66GVIA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.364180013 (383h0m56.826272496s) [noise=true , address=145.79.227.14:4838 , piece_id=S7IRMYZAGLSDRCMMHAGCJ6476AHFMWW4ZKW74IB2G22XWBQVUTSA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:13.299270278 (383h0m58.761362761s) [upload=336995]: Next returned nil segment Oct 9 12:32:13.299290196 (383h0m58.761382679s) [upload=336995]: Waiting for error group managing segments... Oct 9 12:32:13.615010146 (383h0m59.077102629s) [noise=true , address=45.91.116.5:4298 , piece_id=OCGLBI4L424PQ2PFWS4WYMQCVMWPRWDN3IZY5PAB7XV2ESQ52BDA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.646135705 (383h0m59.108228188s) [noise=true , address=45.91.116.11:4118 , piece_id=GW7B7NELLNAI42Z2CJMGGJHATPQHYTWDJALC3TNRX365456B4VLQ, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.725132132 (383h0m59.187224625s) [noise=true , address=77.90.31.15:4697 , piece_id=KEIBEPNKBMQSIVUTUQYTG6RTYRR5CLF2I5PHSTPN7XIWJ7ME26XA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.831029063 (383h0m59.293121546s) [noise=true , address=45.91.116.6:4975 , piece_id=57NQURKND6OPNKNHSKSC4BII5TR2X2WVYJV2UAVN5M7KCSTAHY3A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.844060108 (383h0m59.306152591s) [noise=true , address=77.90.31.17:4772 , piece_id=S5PQWDPOBY7TTFAIYZ3YTYBYVH5CBI5CN5D44ZW2HHUHXIWNGKNA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.924210421 (383h0m59.386302894s) [noise=true , address=45.91.116.7:4009 , piece_id=M7YASOBZYEOLOTHRC3KQ6NOVZZHQENHBM5KWLF54M53CHVXVDB2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.115471763 (383h0m59.577564247s) [noise=true , address=45.91.116.9:4262 , piece_id=LDX4BO2OK6OKZMJZJBDXTVKECZUE22BYIHZII5FL7TGPFKAWDZTA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.137527292 (383h0m59.599619776s) [noise=true , address=145.79.227.13:4803 , piece_id=2BHC3PWTD65IGVGN4653DX36X6RLHXDM5CJSLMJ6L7ZSLREJWT2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.139239683 (383h0m59.601332166s) [noise=true , address=145.79.227.14:4838 , piece_id=S7IRMYZAGLSDRCMMHAGCJ6476AHFMWW4ZKW74IB2G22XWBQVUTSA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.149064161 (383h0m59.611156644s) [noise=true , address=145.79.227.11:4730 , piece_id=4BMR36W56SM2QWQQWD3LRUPZ4GI22NBVOJPEQD5WVRL4LD66GVIA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.156402325 (383h0m59.618494808s) [noise=true , address=145.79.227.5:4222 , piece_id=QPCB2VXUWG22EUQ6WJTEYY42ZDCUGTO6K3OWPQWBOJZJMDBPEWFA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.178407738 (383h0m59.640500241s) [noise=true , address=145.79.227.2:4115 , piece_id=XVKTC2ZKQJL6IREHVUIROSOHMD6YUPAHSQDRSJHP4YIAFJNPWCGA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.246646118 (383h0m59.708738601s) [noise=true , address=145.79.227.3:4150 , piece_id=IAC57SI5LKOJVARZHLCVFIJBK4Q5RYFWP3TR3Z2MFSTUPVSGV53Q, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.246687597 (383h0m59.70878008s) [seg_pos={0 0}, upload=336995]: Segment reached optimal threshold of 13 pieces. Oct 9 12:32:14.246874583 (383h0m59.708967066s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246885854 (383h0m59.708978338s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246910331 (383h0m59.709002824s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246923005 (383h0m59.709015488s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246965816 (383h0m59.7090583s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246975606 (383h0m59.709068089s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246996415 (383h0m59.709088898s) [seg_pos={0 0}, upload=336995]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:32:14.247095574 (383h0m59.709188067s) [upload=336995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:32:14.247103950 (383h0m59.709196433s) [upload=336995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:32:14.670708184 (383h1m0.132800667s) [upload=336995]: Done uploading segments... Oct 9 12:32:14.670733413 (383h1m0.132825896s) [upload=336995]: Upload finished. err: <nil> Duration: 7.579828553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................................X...X...X..XXX...] duration: 7.300226934s FAIL (2025-10-09 12:34:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 12:32:11.014667976 (383h0m56.476760459s) [upload=336995]: Starting upload Oct 9 12:32:11.014753328 (383h0m56.476845811s) [upload=336995]: Done starting upload Oct 9 12:32:11.015033873 (383h0m56.477126366s) [upload=336995]: Uploading segments... Oct 9 12:32:11.015082205 (383h0m56.477174688s) [upload=336995]: Got next segment. Inline: false Oct 9 12:32:11.015122642 (383h0m56.477215125s) [upload=336995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:32:11.015128453 (383h0m56.477220926s) [upload=336995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:32:11.352699541 (383h0m56.814792014s) [seg_pos={0 0}, upload=336995]: Begin upload segment... Oct 9 12:32:11.352754256 (383h0m56.814846739s) [seg_pos={0 0}, upload=336995]: Done begin upload segment. Oct 9 12:32:11.352784163 (383h0m56.814876646s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.353342336 (383h0m56.815434819s) [noise=true , address=45.91.116.11:4118 , piece_id=GW7B7NELLNAI42Z2CJMGGJHATPQHYTWDJALC3TNRX365456B4VLQ, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354029434 (383h0m56.816121927s) [noise=true , address=145.79.227.5:4222 , piece_id=QPCB2VXUWG22EUQ6WJTEYY42ZDCUGTO6K3OWPQWBOJZJMDBPEWFA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354447490 (383h0m56.816539973s) [noise=true , address=45.91.116.5:4298 , piece_id=OCGLBI4L424PQ2PFWS4WYMQCVMWPRWDN3IZY5PAB7XV2ESQ52BDA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354877619 (383h0m56.816970112s) [noise=true , address=145.79.227.13:4803 , piece_id=2BHC3PWTD65IGVGN4653DX36X6RLHXDM5CJSLMJ6L7ZSLREJWT2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.355353105 (383h0m56.817445588s) [noise=true , address=145.79.227.2:4115 , piece_id=XVKTC2ZKQJL6IREHVUIROSOHMD6YUPAHSQDRSJHP4YIAFJNPWCGA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.355789164 (383h0m56.817881647s) [noise=true , address=77.90.31.17:4772 , piece_id=S5PQWDPOBY7TTFAIYZ3YTYBYVH5CBI5CN5D44ZW2HHUHXIWNGKNA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.356805349 (383h0m56.818897832s) [noise=true , address=45.91.116.9:4262 , piece_id=LDX4BO2OK6OKZMJZJBDXTVKECZUE22BYIHZII5FL7TGPFKAWDZTA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.357222053 (383h0m56.819314536s) [noise=true , address=77.90.31.15:4697 , piece_id=KEIBEPNKBMQSIVUTUQYTG6RTYRR5CLF2I5PHSTPN7XIWJ7ME26XA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.358482193 (383h0m56.820574676s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.359022401 (383h0m56.821114894s) [noise=true , address=145.79.227.3:4150 , piece_id=IAC57SI5LKOJVARZHLCVFIJBK4Q5RYFWP3TR3Z2MFSTUPVSGV53Q, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.359922265 (383h0m56.822014748s) [noise=true , address=45.91.116.6:4975 , piece_id=57NQURKND6OPNKNHSKSC4BII5TR2X2WVYJV2UAVN5M7KCSTAHY3A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.360618491 (383h0m56.822710974s) [noise=true , address=45.91.116.7:4009 , piece_id=M7YASOBZYEOLOTHRC3KQ6NOVZZHQENHBM5KWLF54M53CHVXVDB2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.361398246 (383h0m56.823490729s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.362845992 (383h0m56.824938465s) [noise=true , address=145.79.227.11:4730 , piece_id=4BMR36W56SM2QWQQWD3LRUPZ4GI22NBVOJPEQD5WVRL4LD66GVIA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.364180013 (383h0m56.826272496s) [noise=true , address=145.79.227.14:4838 , piece_id=S7IRMYZAGLSDRCMMHAGCJ6476AHFMWW4ZKW74IB2G22XWBQVUTSA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:13.299270278 (383h0m58.761362761s) [upload=336995]: Next returned nil segment Oct 9 12:32:13.299290196 (383h0m58.761382679s) [upload=336995]: Waiting for error group managing segments... Oct 9 12:32:13.615010146 (383h0m59.077102629s) [noise=true , address=45.91.116.5:4298 , piece_id=OCGLBI4L424PQ2PFWS4WYMQCVMWPRWDN3IZY5PAB7XV2ESQ52BDA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.646135705 (383h0m59.108228188s) [noise=true , address=45.91.116.11:4118 , piece_id=GW7B7NELLNAI42Z2CJMGGJHATPQHYTWDJALC3TNRX365456B4VLQ, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.725132132 (383h0m59.187224625s) [noise=true , address=77.90.31.15:4697 , piece_id=KEIBEPNKBMQSIVUTUQYTG6RTYRR5CLF2I5PHSTPN7XIWJ7ME26XA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.831029063 (383h0m59.293121546s) [noise=true , address=45.91.116.6:4975 , piece_id=57NQURKND6OPNKNHSKSC4BII5TR2X2WVYJV2UAVN5M7KCSTAHY3A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.844060108 (383h0m59.306152591s) [noise=true , address=77.90.31.17:4772 , piece_id=S5PQWDPOBY7TTFAIYZ3YTYBYVH5CBI5CN5D44ZW2HHUHXIWNGKNA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.924210421 (383h0m59.386302894s) [noise=true , address=45.91.116.7:4009 , piece_id=M7YASOBZYEOLOTHRC3KQ6NOVZZHQENHBM5KWLF54M53CHVXVDB2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.115471763 (383h0m59.577564247s) [noise=true , address=45.91.116.9:4262 , piece_id=LDX4BO2OK6OKZMJZJBDXTVKECZUE22BYIHZII5FL7TGPFKAWDZTA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.137527292 (383h0m59.599619776s) [noise=true , address=145.79.227.13:4803 , piece_id=2BHC3PWTD65IGVGN4653DX36X6RLHXDM5CJSLMJ6L7ZSLREJWT2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.139239683 (383h0m59.601332166s) [noise=true , address=145.79.227.14:4838 , piece_id=S7IRMYZAGLSDRCMMHAGCJ6476AHFMWW4ZKW74IB2G22XWBQVUTSA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.149064161 (383h0m59.611156644s) [noise=true , address=145.79.227.11:4730 , piece_id=4BMR36W56SM2QWQQWD3LRUPZ4GI22NBVOJPEQD5WVRL4LD66GVIA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.156402325 (383h0m59.618494808s) [noise=true , address=145.79.227.5:4222 , piece_id=QPCB2VXUWG22EUQ6WJTEYY42ZDCUGTO6K3OWPQWBOJZJMDBPEWFA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.178407738 (383h0m59.640500241s) [noise=true , address=145.79.227.2:4115 , piece_id=XVKTC2ZKQJL6IREHVUIROSOHMD6YUPAHSQDRSJHP4YIAFJNPWCGA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.246646118 (383h0m59.708738601s) [noise=true , address=145.79.227.3:4150 , piece_id=IAC57SI5LKOJVARZHLCVFIJBK4Q5RYFWP3TR3Z2MFSTUPVSGV53Q, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.246687597 (383h0m59.70878008s) [seg_pos={0 0}, upload=336995]: Segment reached optimal threshold of 13 pieces. Oct 9 12:32:14.246874583 (383h0m59.708967066s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246885854 (383h0m59.708978338s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246910331 (383h0m59.709002824s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246923005 (383h0m59.709015488s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246965816 (383h0m59.7090583s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246975606 (383h0m59.709068089s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246996415 (383h0m59.709088898s) [seg_pos={0 0}, upload=336995]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:32:14.247095574 (383h0m59.709188067s) [upload=336995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:32:14.247103950 (383h0m59.709196433s) [upload=336995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:32:14.670708184 (383h1m0.132800667s) [upload=336995]: Done uploading segments... Oct 9 12:32:14.670733413 (383h1m0.132825896s) [upload=336995]: Upload finished. err: <nil> Duration: 7.300263162s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................................X...X...X..XXX..] duration: 26.528942157s FAIL (2025-10-09 12:34:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 12:32:11.014667976 (383h0m56.476760459s) [upload=336995]: Starting upload Oct 9 12:32:11.014753328 (383h0m56.476845811s) [upload=336995]: Done starting upload Oct 9 12:32:11.015033873 (383h0m56.477126366s) [upload=336995]: Uploading segments... Oct 9 12:32:11.015082205 (383h0m56.477174688s) [upload=336995]: Got next segment. Inline: false Oct 9 12:32:11.015122642 (383h0m56.477215125s) [upload=336995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:32:11.015128453 (383h0m56.477220926s) [upload=336995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:32:11.352699541 (383h0m56.814792014s) [seg_pos={0 0}, upload=336995]: Begin upload segment... Oct 9 12:32:11.352754256 (383h0m56.814846739s) [seg_pos={0 0}, upload=336995]: Done begin upload segment. Oct 9 12:32:11.352784163 (383h0m56.814876646s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.353342336 (383h0m56.815434819s) [noise=true , address=45.91.116.11:4118 , piece_id=GW7B7NELLNAI42Z2CJMGGJHATPQHYTWDJALC3TNRX365456B4VLQ, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354029434 (383h0m56.816121927s) [noise=true , address=145.79.227.5:4222 , piece_id=QPCB2VXUWG22EUQ6WJTEYY42ZDCUGTO6K3OWPQWBOJZJMDBPEWFA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354447490 (383h0m56.816539973s) [noise=true , address=45.91.116.5:4298 , piece_id=OCGLBI4L424PQ2PFWS4WYMQCVMWPRWDN3IZY5PAB7XV2ESQ52BDA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354877619 (383h0m56.816970112s) [noise=true , address=145.79.227.13:4803 , piece_id=2BHC3PWTD65IGVGN4653DX36X6RLHXDM5CJSLMJ6L7ZSLREJWT2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.355353105 (383h0m56.817445588s) [noise=true , address=145.79.227.2:4115 , piece_id=XVKTC2ZKQJL6IREHVUIROSOHMD6YUPAHSQDRSJHP4YIAFJNPWCGA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.355789164 (383h0m56.817881647s) [noise=true , address=77.90.31.17:4772 , piece_id=S5PQWDPOBY7TTFAIYZ3YTYBYVH5CBI5CN5D44ZW2HHUHXIWNGKNA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.356805349 (383h0m56.818897832s) [noise=true , address=45.91.116.9:4262 , piece_id=LDX4BO2OK6OKZMJZJBDXTVKECZUE22BYIHZII5FL7TGPFKAWDZTA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.357222053 (383h0m56.819314536s) [noise=true , address=77.90.31.15:4697 , piece_id=KEIBEPNKBMQSIVUTUQYTG6RTYRR5CLF2I5PHSTPN7XIWJ7ME26XA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.358482193 (383h0m56.820574676s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.359022401 (383h0m56.821114894s) [noise=true , address=145.79.227.3:4150 , piece_id=IAC57SI5LKOJVARZHLCVFIJBK4Q5RYFWP3TR3Z2MFSTUPVSGV53Q, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.359922265 (383h0m56.822014748s) [noise=true , address=45.91.116.6:4975 , piece_id=57NQURKND6OPNKNHSKSC4BII5TR2X2WVYJV2UAVN5M7KCSTAHY3A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.360618491 (383h0m56.822710974s) [noise=true , address=45.91.116.7:4009 , piece_id=M7YASOBZYEOLOTHRC3KQ6NOVZZHQENHBM5KWLF54M53CHVXVDB2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.361398246 (383h0m56.823490729s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.362845992 (383h0m56.824938465s) [noise=true , address=145.79.227.11:4730 , piece_id=4BMR36W56SM2QWQQWD3LRUPZ4GI22NBVOJPEQD5WVRL4LD66GVIA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.364180013 (383h0m56.826272496s) [noise=true , address=145.79.227.14:4838 , piece_id=S7IRMYZAGLSDRCMMHAGCJ6476AHFMWW4ZKW74IB2G22XWBQVUTSA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:13.299270278 (383h0m58.761362761s) [upload=336995]: Next returned nil segment Oct 9 12:32:13.299290196 (383h0m58.761382679s) [upload=336995]: Waiting for error group managing segments... Oct 9 12:32:13.615010146 (383h0m59.077102629s) [noise=true , address=45.91.116.5:4298 , piece_id=OCGLBI4L424PQ2PFWS4WYMQCVMWPRWDN3IZY5PAB7XV2ESQ52BDA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.646135705 (383h0m59.108228188s) [noise=true , address=45.91.116.11:4118 , piece_id=GW7B7NELLNAI42Z2CJMGGJHATPQHYTWDJALC3TNRX365456B4VLQ, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.725132132 (383h0m59.187224625s) [noise=true , address=77.90.31.15:4697 , piece_id=KEIBEPNKBMQSIVUTUQYTG6RTYRR5CLF2I5PHSTPN7XIWJ7ME26XA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.831029063 (383h0m59.293121546s) [noise=true , address=45.91.116.6:4975 , piece_id=57NQURKND6OPNKNHSKSC4BII5TR2X2WVYJV2UAVN5M7KCSTAHY3A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.844060108 (383h0m59.306152591s) [noise=true , address=77.90.31.17:4772 , piece_id=S5PQWDPOBY7TTFAIYZ3YTYBYVH5CBI5CN5D44ZW2HHUHXIWNGKNA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.924210421 (383h0m59.386302894s) [noise=true , address=45.91.116.7:4009 , piece_id=M7YASOBZYEOLOTHRC3KQ6NOVZZHQENHBM5KWLF54M53CHVXVDB2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.115471763 (383h0m59.577564247s) [noise=true , address=45.91.116.9:4262 , piece_id=LDX4BO2OK6OKZMJZJBDXTVKECZUE22BYIHZII5FL7TGPFKAWDZTA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.137527292 (383h0m59.599619776s) [noise=true , address=145.79.227.13:4803 , piece_id=2BHC3PWTD65IGVGN4653DX36X6RLHXDM5CJSLMJ6L7ZSLREJWT2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.139239683 (383h0m59.601332166s) [noise=true , address=145.79.227.14:4838 , piece_id=S7IRMYZAGLSDRCMMHAGCJ6476AHFMWW4ZKW74IB2G22XWBQVUTSA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.149064161 (383h0m59.611156644s) [noise=true , address=145.79.227.11:4730 , piece_id=4BMR36W56SM2QWQQWD3LRUPZ4GI22NBVOJPEQD5WVRL4LD66GVIA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.156402325 (383h0m59.618494808s) [noise=true , address=145.79.227.5:4222 , piece_id=QPCB2VXUWG22EUQ6WJTEYY42ZDCUGTO6K3OWPQWBOJZJMDBPEWFA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.178407738 (383h0m59.640500241s) [noise=true , address=145.79.227.2:4115 , piece_id=XVKTC2ZKQJL6IREHVUIROSOHMD6YUPAHSQDRSJHP4YIAFJNPWCGA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.246646118 (383h0m59.708738601s) [noise=true , address=145.79.227.3:4150 , piece_id=IAC57SI5LKOJVARZHLCVFIJBK4Q5RYFWP3TR3Z2MFSTUPVSGV53Q, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.246687597 (383h0m59.70878008s) [seg_pos={0 0}, upload=336995]: Segment reached optimal threshold of 13 pieces. Oct 9 12:32:14.246874583 (383h0m59.708967066s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246885854 (383h0m59.708978338s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246910331 (383h0m59.709002824s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246923005 (383h0m59.709015488s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246965816 (383h0m59.7090583s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246975606 (383h0m59.709068089s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246996415 (383h0m59.709088898s) [seg_pos={0 0}, upload=336995]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:32:14.247095574 (383h0m59.709188067s) [upload=336995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:32:14.247103950 (383h0m59.709196433s) [upload=336995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:32:14.670708184 (383h1m0.132800667s) [upload=336995]: Done uploading segments... Oct 9 12:32:14.670733413 (383h1m0.132825896s) [upload=336995]: Upload finished. err: <nil> Duration: 26.52899683s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................................X...X...X..XXX.] duration: 16.002123818s FAIL (2025-10-09 12:32:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 12:32:11.014667976 (383h0m56.476760459s) [upload=336995]: Starting upload Oct 9 12:32:11.014753328 (383h0m56.476845811s) [upload=336995]: Done starting upload Oct 9 12:32:11.015033873 (383h0m56.477126366s) [upload=336995]: Uploading segments... Oct 9 12:32:11.015082205 (383h0m56.477174688s) [upload=336995]: Got next segment. Inline: false Oct 9 12:32:11.015122642 (383h0m56.477215125s) [upload=336995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:32:11.015128453 (383h0m56.477220926s) [upload=336995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:32:11.352699541 (383h0m56.814792014s) [seg_pos={0 0}, upload=336995]: Begin upload segment... Oct 9 12:32:11.352754256 (383h0m56.814846739s) [seg_pos={0 0}, upload=336995]: Done begin upload segment. Oct 9 12:32:11.352784163 (383h0m56.814876646s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.353342336 (383h0m56.815434819s) [noise=true , address=45.91.116.11:4118 , piece_id=GW7B7NELLNAI42Z2CJMGGJHATPQHYTWDJALC3TNRX365456B4VLQ, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354029434 (383h0m56.816121927s) [noise=true , address=145.79.227.5:4222 , piece_id=QPCB2VXUWG22EUQ6WJTEYY42ZDCUGTO6K3OWPQWBOJZJMDBPEWFA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354447490 (383h0m56.816539973s) [noise=true , address=45.91.116.5:4298 , piece_id=OCGLBI4L424PQ2PFWS4WYMQCVMWPRWDN3IZY5PAB7XV2ESQ52BDA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354877619 (383h0m56.816970112s) [noise=true , address=145.79.227.13:4803 , piece_id=2BHC3PWTD65IGVGN4653DX36X6RLHXDM5CJSLMJ6L7ZSLREJWT2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.355353105 (383h0m56.817445588s) [noise=true , address=145.79.227.2:4115 , piece_id=XVKTC2ZKQJL6IREHVUIROSOHMD6YUPAHSQDRSJHP4YIAFJNPWCGA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.355789164 (383h0m56.817881647s) [noise=true , address=77.90.31.17:4772 , piece_id=S5PQWDPOBY7TTFAIYZ3YTYBYVH5CBI5CN5D44ZW2HHUHXIWNGKNA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.356805349 (383h0m56.818897832s) [noise=true , address=45.91.116.9:4262 , piece_id=LDX4BO2OK6OKZMJZJBDXTVKECZUE22BYIHZII5FL7TGPFKAWDZTA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.357222053 (383h0m56.819314536s) [noise=true , address=77.90.31.15:4697 , piece_id=KEIBEPNKBMQSIVUTUQYTG6RTYRR5CLF2I5PHSTPN7XIWJ7ME26XA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.358482193 (383h0m56.820574676s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.359022401 (383h0m56.821114894s) [noise=true , address=145.79.227.3:4150 , piece_id=IAC57SI5LKOJVARZHLCVFIJBK4Q5RYFWP3TR3Z2MFSTUPVSGV53Q, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.359922265 (383h0m56.822014748s) [noise=true , address=45.91.116.6:4975 , piece_id=57NQURKND6OPNKNHSKSC4BII5TR2X2WVYJV2UAVN5M7KCSTAHY3A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.360618491 (383h0m56.822710974s) [noise=true , address=45.91.116.7:4009 , piece_id=M7YASOBZYEOLOTHRC3KQ6NOVZZHQENHBM5KWLF54M53CHVXVDB2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.361398246 (383h0m56.823490729s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.362845992 (383h0m56.824938465s) [noise=true , address=145.79.227.11:4730 , piece_id=4BMR36W56SM2QWQQWD3LRUPZ4GI22NBVOJPEQD5WVRL4LD66GVIA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.364180013 (383h0m56.826272496s) [noise=true , address=145.79.227.14:4838 , piece_id=S7IRMYZAGLSDRCMMHAGCJ6476AHFMWW4ZKW74IB2G22XWBQVUTSA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:13.299270278 (383h0m58.761362761s) [upload=336995]: Next returned nil segment Oct 9 12:32:13.299290196 (383h0m58.761382679s) [upload=336995]: Waiting for error group managing segments... Oct 9 12:32:13.615010146 (383h0m59.077102629s) [noise=true , address=45.91.116.5:4298 , piece_id=OCGLBI4L424PQ2PFWS4WYMQCVMWPRWDN3IZY5PAB7XV2ESQ52BDA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.646135705 (383h0m59.108228188s) [noise=true , address=45.91.116.11:4118 , piece_id=GW7B7NELLNAI42Z2CJMGGJHATPQHYTWDJALC3TNRX365456B4VLQ, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.725132132 (383h0m59.187224625s) [noise=true , address=77.90.31.15:4697 , piece_id=KEIBEPNKBMQSIVUTUQYTG6RTYRR5CLF2I5PHSTPN7XIWJ7ME26XA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.831029063 (383h0m59.293121546s) [noise=true , address=45.91.116.6:4975 , piece_id=57NQURKND6OPNKNHSKSC4BII5TR2X2WVYJV2UAVN5M7KCSTAHY3A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.844060108 (383h0m59.306152591s) [noise=true , address=77.90.31.17:4772 , piece_id=S5PQWDPOBY7TTFAIYZ3YTYBYVH5CBI5CN5D44ZW2HHUHXIWNGKNA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.924210421 (383h0m59.386302894s) [noise=true , address=45.91.116.7:4009 , piece_id=M7YASOBZYEOLOTHRC3KQ6NOVZZHQENHBM5KWLF54M53CHVXVDB2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.115471763 (383h0m59.577564247s) [noise=true , address=45.91.116.9:4262 , piece_id=LDX4BO2OK6OKZMJZJBDXTVKECZUE22BYIHZII5FL7TGPFKAWDZTA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.137527292 (383h0m59.599619776s) [noise=true , address=145.79.227.13:4803 , piece_id=2BHC3PWTD65IGVGN4653DX36X6RLHXDM5CJSLMJ6L7ZSLREJWT2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.139239683 (383h0m59.601332166s) [noise=true , address=145.79.227.14:4838 , piece_id=S7IRMYZAGLSDRCMMHAGCJ6476AHFMWW4ZKW74IB2G22XWBQVUTSA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.149064161 (383h0m59.611156644s) [noise=true , address=145.79.227.11:4730 , piece_id=4BMR36W56SM2QWQQWD3LRUPZ4GI22NBVOJPEQD5WVRL4LD66GVIA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.156402325 (383h0m59.618494808s) [noise=true , address=145.79.227.5:4222 , piece_id=QPCB2VXUWG22EUQ6WJTEYY42ZDCUGTO6K3OWPQWBOJZJMDBPEWFA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.178407738 (383h0m59.640500241s) [noise=true , address=145.79.227.2:4115 , piece_id=XVKTC2ZKQJL6IREHVUIROSOHMD6YUPAHSQDRSJHP4YIAFJNPWCGA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.246646118 (383h0m59.708738601s) [noise=true , address=145.79.227.3:4150 , piece_id=IAC57SI5LKOJVARZHLCVFIJBK4Q5RYFWP3TR3Z2MFSTUPVSGV53Q, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.246687597 (383h0m59.70878008s) [seg_pos={0 0}, upload=336995]: Segment reached optimal threshold of 13 pieces. Oct 9 12:32:14.246874583 (383h0m59.708967066s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246885854 (383h0m59.708978338s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246910331 (383h0m59.709002824s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246923005 (383h0m59.709015488s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246965816 (383h0m59.7090583s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246975606 (383h0m59.709068089s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246996415 (383h0m59.709088898s) [seg_pos={0 0}, upload=336995]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:32:14.247095574 (383h0m59.709188067s) [upload=336995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:32:14.247103950 (383h0m59.709196433s) [upload=336995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:32:14.670708184 (383h1m0.132800667s) [upload=336995]: Done uploading segments... Oct 9 12:32:14.670733413 (383h1m0.132825896s) [upload=336995]: Upload finished. err: <nil> Duration: 16.002145618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................................X...X...X..XXX] duration: 30.002733324s FAIL (2025-10-09 12:32:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 12:32:11.014667976 (383h0m56.476760459s) [upload=336995]: Starting upload Oct 9 12:32:11.014753328 (383h0m56.476845811s) [upload=336995]: Done starting upload Oct 9 12:32:11.015033873 (383h0m56.477126366s) [upload=336995]: Uploading segments... Oct 9 12:32:11.015082205 (383h0m56.477174688s) [upload=336995]: Got next segment. Inline: false Oct 9 12:32:11.015122642 (383h0m56.477215125s) [upload=336995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:32:11.015128453 (383h0m56.477220926s) [upload=336995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:32:11.352699541 (383h0m56.814792014s) [seg_pos={0 0}, upload=336995]: Begin upload segment... Oct 9 12:32:11.352754256 (383h0m56.814846739s) [seg_pos={0 0}, upload=336995]: Done begin upload segment. Oct 9 12:32:11.352784163 (383h0m56.814876646s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.353342336 (383h0m56.815434819s) [noise=true , address=45.91.116.11:4118 , piece_id=GW7B7NELLNAI42Z2CJMGGJHATPQHYTWDJALC3TNRX365456B4VLQ, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354029434 (383h0m56.816121927s) [noise=true , address=145.79.227.5:4222 , piece_id=QPCB2VXUWG22EUQ6WJTEYY42ZDCUGTO6K3OWPQWBOJZJMDBPEWFA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354447490 (383h0m56.816539973s) [noise=true , address=45.91.116.5:4298 , piece_id=OCGLBI4L424PQ2PFWS4WYMQCVMWPRWDN3IZY5PAB7XV2ESQ52BDA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.354877619 (383h0m56.816970112s) [noise=true , address=145.79.227.13:4803 , piece_id=2BHC3PWTD65IGVGN4653DX36X6RLHXDM5CJSLMJ6L7ZSLREJWT2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.355353105 (383h0m56.817445588s) [noise=true , address=145.79.227.2:4115 , piece_id=XVKTC2ZKQJL6IREHVUIROSOHMD6YUPAHSQDRSJHP4YIAFJNPWCGA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.355789164 (383h0m56.817881647s) [noise=true , address=77.90.31.17:4772 , piece_id=S5PQWDPOBY7TTFAIYZ3YTYBYVH5CBI5CN5D44ZW2HHUHXIWNGKNA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.356805349 (383h0m56.818897832s) [noise=true , address=45.91.116.9:4262 , piece_id=LDX4BO2OK6OKZMJZJBDXTVKECZUE22BYIHZII5FL7TGPFKAWDZTA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.357222053 (383h0m56.819314536s) [noise=true , address=77.90.31.15:4697 , piece_id=KEIBEPNKBMQSIVUTUQYTG6RTYRR5CLF2I5PHSTPN7XIWJ7ME26XA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.358482193 (383h0m56.820574676s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.359022401 (383h0m56.821114894s) [noise=true , address=145.79.227.3:4150 , piece_id=IAC57SI5LKOJVARZHLCVFIJBK4Q5RYFWP3TR3Z2MFSTUPVSGV53Q, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.359922265 (383h0m56.822014748s) [noise=true , address=45.91.116.6:4975 , piece_id=57NQURKND6OPNKNHSKSC4BII5TR2X2WVYJV2UAVN5M7KCSTAHY3A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.360618491 (383h0m56.822710974s) [noise=true , address=45.91.116.7:4009 , piece_id=M7YASOBZYEOLOTHRC3KQ6NOVZZHQENHBM5KWLF54M53CHVXVDB2A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.361398246 (383h0m56.823490729s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.362845992 (383h0m56.824938465s) [noise=true , address=145.79.227.11:4730 , piece_id=4BMR36W56SM2QWQQWD3LRUPZ4GI22NBVOJPEQD5WVRL4LD66GVIA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:11.364180013 (383h0m56.826272496s) [noise=true , address=145.79.227.14:4838 , piece_id=S7IRMYZAGLSDRCMMHAGCJ6476AHFMWW4ZKW74IB2G22XWBQVUTSA, seg_pos={0 0}, upload=336995]: Uploading piece... Oct 9 12:32:13.299270278 (383h0m58.761362761s) [upload=336995]: Next returned nil segment Oct 9 12:32:13.299290196 (383h0m58.761382679s) [upload=336995]: Waiting for error group managing segments... Oct 9 12:32:13.615010146 (383h0m59.077102629s) [noise=true , address=45.91.116.5:4298 , piece_id=OCGLBI4L424PQ2PFWS4WYMQCVMWPRWDN3IZY5PAB7XV2ESQ52BDA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.646135705 (383h0m59.108228188s) [noise=true , address=45.91.116.11:4118 , piece_id=GW7B7NELLNAI42Z2CJMGGJHATPQHYTWDJALC3TNRX365456B4VLQ, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.725132132 (383h0m59.187224625s) [noise=true , address=77.90.31.15:4697 , piece_id=KEIBEPNKBMQSIVUTUQYTG6RTYRR5CLF2I5PHSTPN7XIWJ7ME26XA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.831029063 (383h0m59.293121546s) [noise=true , address=45.91.116.6:4975 , piece_id=57NQURKND6OPNKNHSKSC4BII5TR2X2WVYJV2UAVN5M7KCSTAHY3A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.844060108 (383h0m59.306152591s) [noise=true , address=77.90.31.17:4772 , piece_id=S5PQWDPOBY7TTFAIYZ3YTYBYVH5CBI5CN5D44ZW2HHUHXIWNGKNA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:13.924210421 (383h0m59.386302894s) [noise=true , address=45.91.116.7:4009 , piece_id=M7YASOBZYEOLOTHRC3KQ6NOVZZHQENHBM5KWLF54M53CHVXVDB2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.115471763 (383h0m59.577564247s) [noise=true , address=45.91.116.9:4262 , piece_id=LDX4BO2OK6OKZMJZJBDXTVKECZUE22BYIHZII5FL7TGPFKAWDZTA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.137527292 (383h0m59.599619776s) [noise=true , address=145.79.227.13:4803 , piece_id=2BHC3PWTD65IGVGN4653DX36X6RLHXDM5CJSLMJ6L7ZSLREJWT2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.139239683 (383h0m59.601332166s) [noise=true , address=145.79.227.14:4838 , piece_id=S7IRMYZAGLSDRCMMHAGCJ6476AHFMWW4ZKW74IB2G22XWBQVUTSA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.149064161 (383h0m59.611156644s) [noise=true , address=145.79.227.11:4730 , piece_id=4BMR36W56SM2QWQQWD3LRUPZ4GI22NBVOJPEQD5WVRL4LD66GVIA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.156402325 (383h0m59.618494808s) [noise=true , address=145.79.227.5:4222 , piece_id=QPCB2VXUWG22EUQ6WJTEYY42ZDCUGTO6K3OWPQWBOJZJMDBPEWFA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.178407738 (383h0m59.640500241s) [noise=true , address=145.79.227.2:4115 , piece_id=XVKTC2ZKQJL6IREHVUIROSOHMD6YUPAHSQDRSJHP4YIAFJNPWCGA, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.246646118 (383h0m59.708738601s) [noise=true , address=145.79.227.3:4150 , piece_id=IAC57SI5LKOJVARZHLCVFIJBK4Q5RYFWP3TR3Z2MFSTUPVSGV53Q, seg_pos={0 0}, upload=336995]: Done uploading piece. err: <nil> Oct 9 12:32:14.246687597 (383h0m59.70878008s) [seg_pos={0 0}, upload=336995]: Segment reached optimal threshold of 13 pieces. Oct 9 12:32:14.246874583 (383h0m59.708967066s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246885854 (383h0m59.708978338s) [noise=true , address=77.90.31.18:4807 , piece_id=2BLCW4QIASM57565SUEFIOVGEU2C4PJAYHANET3DV3S7HZSVGX2A, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246910331 (383h0m59.709002824s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246923005 (383h0m59.709015488s) [noise=true , address=145.79.227.18:4006 , piece_id=AXLFDEI4R2PSTOSCPIX6KDVDS4F5NG4UYF3SACYJOUW2A3SWHF7A, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246965816 (383h0m59.7090583s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Oct 9 12:32:14.246975606 (383h0m59.709068089s) [noise=true , address=77.90.31.21:4878 , piece_id=YRK225VKPPCUKQHC7ACUSBC2O5M73N4JX4EWM6MRCLVUTF2WCOEQ, seg_pos={0 0}, upload=336995]: Long tail cancellation due to optimal threshold Oct 9 12:32:14.246996415 (383h0m59.709088898s) [seg_pos={0 0}, upload=336995]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:32:14.247095574 (383h0m59.709188067s) [upload=336995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:32:14.247103950 (383h0m59.709196433s) [upload=336995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:32:14.670708184 (383h1m0.132800667s) [upload=336995]: Done uploading segments... Oct 9 12:32:14.670733413 (383h1m0.132825896s) [upload=336995]: Upload finished. err: <nil> Duration: 30.00274645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................................X...X...X..XX] duration: 30.003085206s FAIL (2025-10-09 12:32:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: BGGA7D3QN2GABZDBZ7CQUID4WZTEQ2DVI5QEZZGVDZL5ZC3ZFW6A) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: RQUUYC26AIKG5ZFB6TYQH27YL4SZ53ZQKJ4RYBOZCBKSQY76VHKQ) context deadline exceeded ======LOGS====== Oct 9 12:31:40.796009180 (383h0m26.258101663s) [upload=336986]: Starting upload Oct 9 12:31:40.796121905 (383h0m26.258214388s) [upload=336986]: Done starting upload Oct 9 12:31:40.796239399 (383h0m26.258331882s) [upload=336986]: Uploading segments... Oct 9 12:31:40.796375378 (383h0m26.258467861s) [upload=336986]: Got next segment. Inline: false Oct 9 12:31:40.796431715 (383h0m26.258524198s) [upload=336986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:31:40.796543468 (383h0m26.258635951s) [upload=336986]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:31:41.048869181 (383h0m26.510961694s) [seg_pos={0 0}, upload=336986]: Begin upload segment... Oct 9 12:31:41.048926821 (383h0m26.511019304s) [seg_pos={0 0}, upload=336986]: Done begin upload segment. Oct 9 12:31:41.048963331 (383h0m26.511055814s) [noise=true , address=45.91.116.10:4081 , piece_id=UPDP6KJCCJKWTENSF7HWBBEHVITN4I273QB5PLQ5DSR5XNQPBH2A, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.049404180 (383h0m26.511496653s) [noise=true , address=77.90.31.18:4805 , piece_id=BGGA7D3QN2GABZDBZ7CQUID4WZTEQ2DVI5QEZZGVDZL5ZC3ZFW6A, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.049922426 (383h0m26.51201491s) [noise=true , address=45.91.116.8:4045 , piece_id=RQUUYC26AIKG5ZFB6TYQH27YL4SZ53ZQKJ4RYBOZCBKSQY76VHKQ, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.050347326 (383h0m26.512439809s) [noise=true , address=145.79.227.6:4257 , piece_id=LX6WEUDN26FJAVZZNJTPMJXOLCAMMTPYS5POZAQFDTQF6CO7SUMQ, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.050747638 (383h0m26.512840121s) [noise=true , address=77.90.31.19:4841 , piece_id=WTBZQPJZTON5XJNDOB6IMKKG6DAYLTRARRQPVW5CAMY6QR25QA6Q, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.051167728 (383h0m26.513260211s) [noise=true , address=145.79.227.2:4115 , piece_id=GW6XJO4K7LZ3HA5HZYRS5VIW6VGOWJZEOGNDREXEVCIRPEEPJLBQ, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.051618997 (383h0m26.51371149s) [noise=true , address=45.91.116.5:4297 , piece_id=KG4Z3KORLXDCGQ7M7ZTXXGRV77HWO4BTQGKOEXVWOPTMARZW2I5A, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.052082429 (383h0m26.514174913s) [noise=true , address=45.91.116.12:4225 , piece_id=MC2RMNMNRCTD34VVYNGOPD7D7GJLLKDRAMZBYFRAYGXAHQ45EZRA, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.053601713 (383h0m26.515694196s) [noise=true , address=45.91.116.7:4011 , piece_id=EGVN7E5Y2XOCMIF5Q6LVEPKXG63DVGCYH67JOI5CCFBCB3RCVYWQ, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.053784390 (383h0m26.515876873s) [noise=true , address=145.79.227.19:4078 , piece_id=7YRUY2ZTBZRZE6GOBRTX767F7VVXVGZLK6GWCM327FITSCWX7SFA, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.055141575 (383h0m26.517234058s) [noise=true , address=45.91.116.4:4938 , piece_id=IK6VLZ3VXMRWDTILDREOJYNOTQ2NZJRCMTDMH7AWEAEYGKMN55QA, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.056641160 (383h0m26.518733633s) [noise=true , address=145.79.227.3:4150 , piece_id=AGWHMIF4ZIEMA3AHC4F4I6ULVTG7SCMHIBFBCGKPL7SZF7VGVK3Q, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.058036819 (383h0m26.520129292s) [noise=true , address=145.79.227.5:4221 , piece_id=J7SL4WOFHP27YAWVJPJ62YQXJMQA2UMTT3GCGZNNE3YOYGEGR2IQ, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.059372602 (383h0m26.521465085s) [noise=true , address=145.79.227.18:4007 , piece_id=MLOBN4DHWML27HUF3H4FITUFWUYL5BUZYB3XFZJWG4ZFD33ZH4OA, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.060824627 (383h0m26.52291713s) [noise=true , address=145.79.227.15:4873 , piece_id=7RYVLY2RSBE3QOYOOXBUGFVCRUFLL5KPB3AQMBQ4SNWACBK7WVYA, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:41.061567922 (383h0m26.523660415s) [noise=true , address=145.79.227.12:4767 , piece_id=2P7Z7CLLJDSXQ7F3A2DEU5WK4VTURVMWCALF6Y2OC3OZ66TZ7BCA, seg_pos={0 0}, upload=336986]: Uploading piece... Oct 9 12:31:42.938528105 (383h0m28.400620608s) [upload=336986]: Next returned nil segment Oct 9 12:31:42.938551729 (383h0m28.400644232s) [upload=336986]: Waiting for error group managing segments... Oct 9 12:31:43.238473065 (383h0m28.700565548s) [noise=true , address=45.91.116.12:4225 , piece_id=MC2RMNMNRCTD34VVYNGOPD7D7GJLLKDRAMZBYFRAYGXAHQ45EZRA, seg_pos={0 0}, upload=336986]: Done uploading piece. err: <nil> Oct 9 12:31:43.314656144 (383h0m28.776748627s) [noise=true , address=45.91.116.10:4081 , piece_id=UPDP6KJCCJKWTENSF7HWBBEHVITN4I273QB5PLQ5DSR5XNQPBH2A, seg_pos={0 0}, upload=336986]: Done uploading piece. err: <nil> Oct 9 12:31:43.351115891 (383h0m28.813208374s) [noise=true , address=45.91.116.5:4297 , piece_id=KG4Z3KORLXDCGQ7M7ZTXXGRV77HWO4BTQGKOEXVWOPTMARZW2I5A, seg_pos={0 0}, upload=336986]: Done uploading piece. err: <nil> Oct 9 12:31:43.397460562 (383h0m28.859553045s) [noise=true , address=45.91.116.4:4938 , piece_id=IK6VLZ3VXMRWDTILDREOJYNOTQ2NZJRCMTDMH7AWEAEYGKMN55QA, seg_pos={0 0}, upload=336986]: Done uploading piece. err: <nil> Oct 9 12:31:43.410046929 (383h0m28.872139402s) [noise=true , address=77.90.31.19:4841 , piece_id=WTBZQPJZTON5XJNDOB6IMKKG6DAYLTRARRQPVW5CAMY6QR25QA6Q, seg_pos={0 0}, upload=336986]: Done uploading piece. err: <nil> Oct 9 12:31:43.430387842 (383h0m28.892480325s) [noise=true , address=77.90.31.18:4805 , piece_id=BGGA7D3QN2GABZDBZ7CQUID4WZTEQ2DVI5QEZZGVDZL5ZC3ZFW6A, seg_pos={0 0}, upload=336986]: Done uploading piece. err: <nil> Oct 9 12:31:43.458751172 (383h0m28.920843665s) [noise=true , address=45.91.116.7:4011 , piece_id=EGVN7E5Y2XOCMIF5Q6LVEPKXG63DVGCYH67JOI5CCFBCB3RCVYWQ, seg_pos={0 0}, upload=336986]: Done uploading piece. err: <nil> Oct 9 12:31:43.475709284 (383h0m28.937801767s) [noise=true , address=45.91.116.8:4045 , piece_id=RQUUYC26AIKG5ZFB6TYQH27YL4SZ53ZQKJ4RYBOZCBKSQY76VHKQ, seg_pos={0 0}, upload=336986]: Done uploading piece. err: <nil> Oct 9 12:31:43.770875455 (383h0m29.232967928s) [noise=true , address=145.79.227.15:4873 , piece_id=7RYVLY2RSBE3QOYOOXBUGFVCRUFLL5KPB3AQMBQ4SNWACBK7WVYA, seg_pos={0 0}, upload=336986]: Done uploading piece. err: <nil> Oct 9 12:31:43.786732491 (383h0m29.248824974s) [noise=true , address=145.79.227.5:4221 , piece_id=J7SL4WOFHP27YAWVJPJ62YQXJMQA2UMTT3GCGZNNE3YOYGEGR2IQ, seg_pos={0 0}, upload=336986]: Done uploading piece. err: <nil> Oct 9 12:31:43.811621896 (383h0m29.273714379s) [noise=true , address=145.79.227.12:4767 , piece_id=2P7Z7CLLJDSXQ7F3A2DEU5WK4VTURVMWCALF6Y2OC3OZ66TZ7BCA, seg_pos={0 0}, upload=336986]: Done uploading piece. err: <nil> Oct 9 12:31:43.816116975 (383h0m29.278209458s) [noise=true , address=145.79.227.19:4078 , piece_id=7YRUY2ZTBZRZE6GOBRTX767F7VVXVGZLK6GWCM327FITSCWX7SFA, seg_pos={0 0}, upload=336986]: Done uploading piece. err: <nil> Oct 9 12:31:43.850900892 (383h0m29.312993375s) [noise=true , address=145.79.227.2:4115 , piece_id=GW6XJO4K7LZ3HA5HZYRS5VIW6VGOWJZEOGNDREXEVCIRPEEPJLBQ, seg_pos={0 0}, upload=336986]: Done uploading piece. err: <nil> Oct 9 12:31:43.850953021 (383h0m29.313045495s) [seg_pos={0 0}, upload=336986]: Segment reached optimal threshold of 13 pieces. Oct 9 12:31:43.851177689 (383h0m29.313270172s) [noise=true , address=145.79.227.18:4007 , piece_id=MLOBN4DHWML27HUF3H4FITUFWUYL5BUZYB3XFZJWG4ZFD33ZH4OA, seg_pos={0 0}, upload=336986]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Oct 9 12:31:43.851189982 (383h0m29.313282485s) [noise=true , address=145.79.227.18:4007 , piece_id=MLOBN4DHWML27HUF3H4FITUFWUYL5BUZYB3XFZJWG4ZFD33ZH4OA, seg_pos={0 0}, upload=336986]: Long tail cancellation due to optimal threshold Oct 9 12:31:43.851271738 (383h0m29.313364211s) [noise=true , address=145.79.227.3:4150 , piece_id=AGWHMIF4ZIEMA3AHC4F4I6ULVTG7SCMHIBFBCGKPL7SZF7VGVK3Q, seg_pos={0 0}, upload=336986]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 12:31:43.851278501 (383h0m29.313370974s) [noise=true , address=145.79.227.3:4150 , piece_id=AGWHMIF4ZIEMA3AHC4F4I6ULVTG7SCMHIBFBCGKPL7SZF7VGVK3Q, seg_pos={0 0}, upload=336986]: Long tail cancellation due to optimal threshold Oct 9 12:31:43.851359264 (383h0m29.313451737s) [noise=true , address=145.79.227.6:4257 , piece_id=LX6WEUDN26FJAVZZNJTPMJXOLCAMMTPYS5POZAQFDTQF6CO7SUMQ, seg_pos={0 0}, upload=336986]: Done uploading piece. err: <nil> Oct 9 12:31:43.851406385 (383h0m29.313498868s) [seg_pos={0 0}, upload=336986]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Oct 9 12:31:43.851516154 (383h0m29.313608637s) [upload=336986]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:31:43.851524870 (383h0m29.313617343s) [upload=336986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:31:44.178784342 (383h0m29.640876815s) [upload=336986]: Done uploading segments... Oct 9 12:31:44.178812787 (383h0m29.64090528s) [upload=336986]: Upload finished. err: <nil> Duration: 30.003125011s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................X...X...X..X] duration: 30.002438383s FAIL (2025-10-09 12:31:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 4ZZ63VFCJL64IKWA2XERPRFOLNDIXLFV4KPQFJ5XG7CG74W3LJSA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: AEVURTEDZEAOXLXPOR4EFKNXTHFKTR3UO7HXIEDNC3JPZLLYC7OA) context deadline exceeded ======LOGS====== Oct 9 12:31:04.118998792 (382h59m49.581091275s) [upload=336975]: Starting upload Oct 9 12:31:04.120828807 (382h59m49.5829213s) [upload=336975]: Done starting upload Oct 9 12:31:04.121132925 (382h59m49.583225408s) [upload=336975]: Uploading segments... Oct 9 12:31:04.121188291 (382h59m49.583280774s) [upload=336975]: Got next segment. Inline: false Oct 9 12:31:04.121229409 (382h59m49.583321892s) [upload=336975]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:31:04.121235591 (382h59m49.583328074s) [upload=336975]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:31:04.371606492 (382h59m49.833698975s) [seg_pos={0 0}, upload=336975]: Begin upload segment... Oct 9 12:31:04.371674531 (382h59m49.833767014s) [seg_pos={0 0}, upload=336975]: Done begin upload segment. Oct 9 12:31:04.371703617 (382h59m49.83379611s) [noise=true , address=77.90.31.11:4555 , piece_id=V2LP2VIX6EXOIJMVUR5WSEIYCR7F2AKOCDYTUGOA2IRDHYZYGU7A, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.372880277 (382h59m49.83497275s) [noise=true , address=145.79.227.2:4114 , piece_id=CDGLUQXLKT3XUKGJ5YHLQKN7OT65UJ4WDB6O5ZLKJVWKW7H2NXDQ, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.373336436 (382h59m49.835428929s) [noise=true , address=45.91.116.13:4153 , piece_id=6GZO5WMDECXIZV4AJ6FYQ66WFL4XBLDVKFCFQQKIDM6FWCJ2KRRA, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.373730546 (382h59m49.835823029s) [noise=true , address=45.91.116.7:4010 , piece_id=N7ZVX7IDEGFQRVFO4TXWHSNEPP5CE6ONFMU2O3TYVJDHNXLPLXLA, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.374187436 (382h59m49.836279919s) [noise=true , address=145.79.227.18:4006 , piece_id=7AQXP5RDTDANQ6S7SY7PWVBH33MXOFFFBJH2JLAK7X6F3KHDTWEQ, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.374597477 (382h59m49.83668996s) [noise=true , address=45.91.116.14:4189 , piece_id=S6RWWGDMLEEDSR2VALO3OHTWLLCJ32YNGSYQVQ6DHEL4YM7FSDRA, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.375078914 (382h59m49.837171397s) [noise=true , address=45.91.116.10:4083 , piece_id=O6OVE2QITG4FLDOQYX34EW3JGSJJCT4VNYMU7EG233YH3TLIC25Q, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.376005337 (382h59m49.83809782s) [noise=true , address=145.79.227.13:4801 , piece_id=HHUPS77BZUVKWZ6UWXH3NC737L3N5CZVYLXBUSYQPOKOWZSWUDSQ, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.376748924 (382h59m49.838841407s) [noise=true , address=77.90.31.19:4843 , piece_id=X6OFP5W652TZXFAGLENXMQZPXOOHLZDMK34MMKE4BLQA3Z7R2SHA, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.377569016 (382h59m49.839661499s) [noise=true , address=145.79.227.17:4969 , piece_id=F6EVRX3HMHJWQFJZHWVPICKUW4QIWSRBF4DHMRXHINWHES4OBLWQ, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.380570962 (382h59m49.842663465s) [noise=true , address=145.79.227.15:4875 , piece_id=5QBXIPZ44ZOC7YWQPLHFI7YPINACD23OMHVTRASWUHBQSYJZNM3Q, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.384049105 (382h59m49.846141598s) [noise=true , address=145.79.227.14:4837 , piece_id=3MZKYAEF7JFA52WJJGFZ4DB5MT63R5UWQCLXHL5N6TVA4O4MXNTQ, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.388620479 (382h59m49.850712962s) [noise=true , address=145.79.227.3:4151 , piece_id=NLHJPMESYSR7WGK7HS4MILKCOYUXH37HFCGVXBTIKD6HJ6JISDMA, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.391239106 (382h59m49.85333159s) [noise=true , address=145.79.227.5:4223 , piece_id=VNJWVB7MHDXYPBYOLF5RPQCIB3ZEX4KG3GQOD23U263YED5GAFKQ, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.394411026 (382h59m49.856503509s) [noise=true , address=77.90.31.18:4805 , piece_id=4ZZ63VFCJL64IKWA2XERPRFOLNDIXLFV4KPQFJ5XG7CG74W3LJSA, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:04.396480648 (382h59m49.858573151s) [noise=true , address=77.90.31.12:4589 , piece_id=AEVURTEDZEAOXLXPOR4EFKNXTHFKTR3UO7HXIEDNC3JPZLLYC7OA, seg_pos={0 0}, upload=336975]: Uploading piece... Oct 9 12:31:06.213129314 (382h59m51.675221807s) [upload=336975]: Next returned nil segment Oct 9 12:31:06.213157928 (382h59m51.675250412s) [upload=336975]: Waiting for error group managing segments... Oct 9 12:31:06.641749471 (382h59m52.103841954s) [noise=true , address=45.91.116.13:4153 , piece_id=6GZO5WMDECXIZV4AJ6FYQ66WFL4XBLDVKFCFQQKIDM6FWCJ2KRRA, seg_pos={0 0}, upload=336975]: Done uploading piece. err: <nil> Oct 9 12:31:06.662691859 (382h59m52.124784342s) [noise=true , address=45.91.116.14:4189 , piece_id=S6RWWGDMLEEDSR2VALO3OHTWLLCJ32YNGSYQVQ6DHEL4YM7FSDRA, seg_pos={0 0}, upload=336975]: Done uploading piece. err: <nil> Oct 9 12:31:06.678561929 (382h59m52.140654413s) [noise=true , address=77.90.31.11:4555 , piece_id=V2LP2VIX6EXOIJMVUR5WSEIYCR7F2AKOCDYTUGOA2IRDHYZYGU7A, seg_pos={0 0}, upload=336975]: Done uploading piece. err: <nil> Oct 9 12:31:06.697703458 (382h59m52.159795931s) [noise=true , address=77.90.31.12:4589 , piece_id=AEVURTEDZEAOXLXPOR4EFKNXTHFKTR3UO7HXIEDNC3JPZLLYC7OA, seg_pos={0 0}, upload=336975]: Done uploading piece. err: <nil> Oct 9 12:31:06.794692156 (382h59m52.256784639s) [noise=true , address=77.90.31.18:4805 , piece_id=4ZZ63VFCJL64IKWA2XERPRFOLNDIXLFV4KPQFJ5XG7CG74W3LJSA, seg_pos={0 0}, upload=336975]: Done uploading piece. err: <nil> Oct 9 12:31:06.859171349 (382h59m52.321263822s) [noise=true , address=45.91.116.10:4083 , piece_id=O6OVE2QITG4FLDOQYX34EW3JGSJJCT4VNYMU7EG233YH3TLIC25Q, seg_pos={0 0}, upload=336975]: Done uploading piece. err: <nil> Oct 9 12:31:06.997469578 (382h59m52.459562062s) [noise=true , address=77.90.31.19:4843 , piece_id=X6OFP5W652TZXFAGLENXMQZPXOOHLZDMK34MMKE4BLQA3Z7R2SHA, seg_pos={0 0}, upload=336975]: Done uploading piece. err: <nil> Oct 9 12:31:07.153521565 (382h59m52.615614059s) [noise=true , address=145.79.227.18:4006 , piece_id=7AQXP5RDTDANQ6S7SY7PWVBH33MXOFFFBJH2JLAK7X6F3KHDTWEQ, seg_pos={0 0}, upload=336975]: Done uploading piece. err: <nil> Oct 9 12:31:07.170985251 (382h59m52.633077744s) [noise=true , address=145.79.227.15:4875 , piece_id=5QBXIPZ44ZOC7YWQPLHFI7YPINACD23OMHVTRASWUHBQSYJZNM3Q, seg_pos={0 0}, upload=336975]: Done uploading piece. err: <nil> Oct 9 12:31:07.175800479 (382h59m52.637892952s) [noise=true , address=145.79.227.17:4969 , piece_id=F6EVRX3HMHJWQFJZHWVPICKUW4QIWSRBF4DHMRXHINWHES4OBLWQ, seg_pos={0 0}, upload=336975]: Done uploading piece. err: <nil> Oct 9 12:31:07.182480681 (382h59m52.644573154s) [noise=true , address=145.79.227.5:4223 , piece_id=VNJWVB7MHDXYPBYOLF5RPQCIB3ZEX4KG3GQOD23U263YED5GAFKQ, seg_pos={0 0}, upload=336975]: Done uploading piece. err: <nil> Oct 9 12:31:07.194969732 (382h59m52.657062215s) [noise=true , address=145.79.227.3:4151 , piece_id=NLHJPMESYSR7WGK7HS4MILKCOYUXH37HFCGVXBTIKD6HJ6JISDMA, seg_pos={0 0}, upload=336975]: Done uploading piece. err: <nil> Oct 9 12:31:07.249263048 (382h59m52.711355522s) [noise=true , address=145.79.227.13:4801 , piece_id=HHUPS77BZUVKWZ6UWXH3NC737L3N5CZVYLXBUSYQPOKOWZSWUDSQ, seg_pos={0 0}, upload=336975]: Done uploading piece. err: <nil> Oct 9 12:31:07.249304728 (382h59m52.711397211s) [seg_pos={0 0}, upload=336975]: Segment reached optimal threshold of 13 pieces. Oct 9 12:31:07.249937222 (382h59m52.712029705s) [noise=true , address=145.79.227.14:4837 , piece_id=3MZKYAEF7JFA52WJJGFZ4DB5MT63R5UWQCLXHL5N6TVA4O4MXNTQ, seg_pos={0 0}, upload=336975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Oct 9 12:31:07.249952091 (382h59m52.712044574s) [noise=true , address=145.79.227.14:4837 , piece_id=3MZKYAEF7JFA52WJJGFZ4DB5MT63R5UWQCLXHL5N6TVA4O4MXNTQ, seg_pos={0 0}, upload=336975]: Long tail cancellation due to optimal threshold Oct 9 12:31:07.250137774 (382h59m52.712230267s) [noise=true , address=145.79.227.2:4114 , piece_id=CDGLUQXLKT3XUKGJ5YHLQKN7OT65UJ4WDB6O5ZLKJVWKW7H2NXDQ, seg_pos={0 0}, upload=336975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: expected piece hash; context canceled; EOF Oct 9 12:31:07.250147383 (382h59m52.712239866s) [noise=true , address=145.79.227.2:4114 , piece_id=CDGLUQXLKT3XUKGJ5YHLQKN7OT65UJ4WDB6O5ZLKJVWKW7H2NXDQ, seg_pos={0 0}, upload=336975]: Long tail cancellation due to optimal threshold Oct 9 12:31:07.250381738 (382h59m52.712474222s) [noise=true , address=45.91.116.7:4010 , piece_id=N7ZVX7IDEGFQRVFO4TXWHSNEPP5CE6ONFMU2O3TYVJDHNXLPLXLA, seg_pos={0 0}, upload=336975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:40010->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 9 12:31:07.250390456 (382h59m52.712482939s) [noise=true , address=45.91.116.7:4010 , piece_id=N7ZVX7IDEGFQRVFO4TXWHSNEPP5CE6ONFMU2O3TYVJDHNXLPLXLA, seg_pos={0 0}, upload=336975]: Long tail cancellation due to optimal threshold Oct 9 12:31:07.250409792 (382h59m52.712502275s) [seg_pos={0 0}, upload=336975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:31:07.250501707 (382h59m52.712594191s) [upload=336975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:31:07.250509342 (382h59m52.712601815s) [upload=336975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:31:07.567944358 (382h59m53.030036851s) [upload=336975]: Done uploading segments... Oct 9 12:31:07.567968083 (382h59m53.030060567s) [upload=336975]: Upload finished. err: <nil> Duration: 30.002465345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X...X...X..] duration: 26.842803259s FAIL (2025-10-09 12:30:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 2E2ERXEHZ7ETMSL3JG26LDAK75SNEQD7PRGBILIUAQFJWTWYPGDA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: NLMPTFLKFV4CBW5MUGRNM65257PXLDOSNS23O34NSG7K6XV7X3XQ) context deadline exceeded ======LOGS====== Oct 9 12:28:51.645526499 (382h57m37.107618982s) [upload=336945]: Starting upload Oct 9 12:28:51.645626680 (382h57m37.107719173s) [upload=336945]: Done starting upload Oct 9 12:28:51.645903016 (382h57m37.107995499s) [upload=336945]: Uploading segments... Oct 9 12:28:51.646109018 (382h57m37.108201501s) [upload=336945]: Got next segment. Inline: false Oct 9 12:28:51.646190253 (382h57m37.108282736s) [upload=336945]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:28:51.646197367 (382h57m37.10828985s) [upload=336945]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:28:51.895659356 (382h57m37.357751839s) [seg_pos={0 0}, upload=336945]: Begin upload segment... Oct 9 12:28:51.895738718 (382h57m37.357831201s) [seg_pos={0 0}, upload=336945]: Done begin upload segment. Oct 9 12:28:51.895771059 (382h57m37.357863542s) [noise=true , address=45.91.116.5:4299 , piece_id=2E2ERXEHZ7ETMSL3JG26LDAK75SNEQD7PRGBILIUAQFJWTWYPGDA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.896756075 (382h57m37.358848558s) [noise=true , address=77.90.31.10:4519 , piece_id=P5BLQQF6JDOFB5RRICRA4BCHYABXWK7JKCODDONPO5KNY5LUBKMQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.897246589 (382h57m37.359339072s) [noise=true , address=45.91.116.14:4190 , piece_id=426KBG5VMF3X76DSOEGTM6NPGXG5BDNRXIVY2DRFK234ZBJPA5FA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.897703198 (382h57m37.359795682s) [noise=true , address=45.91.116.11:4118 , piece_id=NLMPTFLKFV4CBW5MUGRNM65257PXLDOSNS23O34NSG7K6XV7X3XQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.898150530 (382h57m37.360243014s) [noise=true , address=145.79.227.4:4185 , piece_id=WZGAAFLQ2UHAWQ7XJXKUATXAYMFJTIQ57OYZGBPZAXCOBFKW3WSQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.898609644 (382h57m37.360702128s) [noise=true , address=145.79.227.5:4222 , piece_id=3LPHHZJ7PIO3XC4QFER3HUQUKGRDKWB2WO4N3LKWP64FYS6YJGAQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.899094027 (382h57m37.36118651s) [noise=true , address=145.79.227.3:4150 , piece_id=4US3ZYUPQNOT4CHJMUUTCFIEIECELNIIRDDVLO4Z7NTN7L7QE3EQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.899566788 (382h57m37.361659271s) [noise=true , address=145.79.227.2:4114 , piece_id=RWVGXJNEZSGSK2W6BCHEMJL3QWTMYM3ZYRPPA4CZMNV5B2KJC2EQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.900128828 (382h57m37.362221311s) [noise=true , address=145.79.227.12:4767 , piece_id=75IDFDIKKU7S22O4D6JU3ILAOQVPN7GCAJMHOYLG4JD43M3RKW5A, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.900630332 (382h57m37.362722805s) [noise=true , address=145.79.227.15:4874 , piece_id=Z34KEBHVDIKNBGKZOQHNDLRWY6FIHDDP6DXG2RLN2WIBWL2MMJZA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.900960972 (382h57m37.363053455s) [noise=true , address=145.79.227.17:4969 , piece_id=Z7NELFATM54L6M7TN3QZY47F54Z5H76KMRJZCRELLSRM4G7M27WA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.901361484 (382h57m37.363453967s) [noise=true , address=145.79.227.18:4006 , piece_id=FMUIMEPSKKC6C5RWKN6YLNLYOUDBCRS2CIXGHVA6MZWCZXFW6JMA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.901680702 (382h57m37.363773185s) [noise=true , address=145.79.227.5:4223 , piece_id=EWICQ5IHQCEMXX4KQU6LADIB6Z54SSJ7JL4JNGBREMYQAGMULPPA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.905970862 (382h57m37.368063335s) [noise=true , address=145.79.227.11:4731 , piece_id=XHJB4F22MQ4RSAL54XIJYRH7M5JEKFKNIJ4SCPRVRVWMCXXPNBAQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.908040112 (382h57m37.370132585s) [noise=true , address=145.79.227.7:4295 , piece_id=EES54AYYAOIE6E67LRO3QBBOK63AANZ2U6RRHTFZBLNXVW2672JA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.910078714 (382h57m37.372171197s) [noise=true , address=45.91.116.8:4048 , piece_id=DU2ML2LKNW734TWWW3EXM6PLMI23NOGVCQR22V6AQ4YFNE67QVRA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:53.880123120 (382h57m39.342215603s) [upload=336945]: Next returned nil segment Oct 9 12:28:53.880146174 (382h57m39.342238657s) [upload=336945]: Waiting for error group managing segments... Oct 9 12:28:54.397570323 (382h57m39.859662796s) [noise=true , address=45.91.116.8:4048 , piece_id=DU2ML2LKNW734TWWW3EXM6PLMI23NOGVCQR22V6AQ4YFNE67QVRA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.427819273 (382h57m39.889911746s) [noise=true , address=145.79.227.7:4295 , piece_id=EES54AYYAOIE6E67LRO3QBBOK63AANZ2U6RRHTFZBLNXVW2672JA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.434534380 (382h57m39.896626873s) [noise=true , address=45.91.116.11:4118 , piece_id=NLMPTFLKFV4CBW5MUGRNM65257PXLDOSNS23O34NSG7K6XV7X3XQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.435744584 (382h57m39.897837068s) [noise=true , address=145.79.227.15:4874 , piece_id=Z34KEBHVDIKNBGKZOQHNDLRWY6FIHDDP6DXG2RLN2WIBWL2MMJZA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.436162019 (382h57m39.898254492s) [noise=true , address=145.79.227.11:4731 , piece_id=XHJB4F22MQ4RSAL54XIJYRH7M5JEKFKNIJ4SCPRVRVWMCXXPNBAQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.465006876 (382h57m39.927099359s) [noise=true , address=45.91.116.5:4299 , piece_id=2E2ERXEHZ7ETMSL3JG26LDAK75SNEQD7PRGBILIUAQFJWTWYPGDA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.478108986 (382h57m39.940201469s) [noise=true , address=45.91.116.14:4190 , piece_id=426KBG5VMF3X76DSOEGTM6NPGXG5BDNRXIVY2DRFK234ZBJPA5FA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.481253283 (382h57m39.943345756s) [noise=true , address=145.79.227.12:4767 , piece_id=75IDFDIKKU7S22O4D6JU3ILAOQVPN7GCAJMHOYLG4JD43M3RKW5A, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.485721652 (382h57m39.947814135s) [noise=true , address=145.79.227.17:4969 , piece_id=Z7NELFATM54L6M7TN3QZY47F54Z5H76KMRJZCRELLSRM4G7M27WA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.486622207 (382h57m39.94871468s) [noise=true , address=145.79.227.5:4223 , piece_id=EWICQ5IHQCEMXX4KQU6LADIB6Z54SSJ7JL4JNGBREMYQAGMULPPA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.493042041 (382h57m39.955134524s) [noise=true , address=145.79.227.4:4185 , piece_id=WZGAAFLQ2UHAWQ7XJXKUATXAYMFJTIQ57OYZGBPZAXCOBFKW3WSQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.493683773 (382h57m39.955776246s) [noise=true , address=145.79.227.2:4114 , piece_id=RWVGXJNEZSGSK2W6BCHEMJL3QWTMYM3ZYRPPA4CZMNV5B2KJC2EQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.531421124 (382h57m39.993513607s) [noise=true , address=77.90.31.10:4519 , piece_id=P5BLQQF6JDOFB5RRICRA4BCHYABXWK7JKCODDONPO5KNY5LUBKMQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.531455710 (382h57m39.993548193s) [seg_pos={0 0}, upload=336945]: Segment reached optimal threshold of 13 pieces. Oct 9 12:28:54.531610754 (382h57m39.993703237s) [noise=true , address=145.79.227.5:4222 , piece_id=3LPHHZJ7PIO3XC4QFER3HUQUKGRDKWB2WO4N3LKWP64FYS6YJGAQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Oct 9 12:28:54.531623670 (382h57m39.993716153s) [noise=true , address=145.79.227.5:4222 , piece_id=3LPHHZJ7PIO3XC4QFER3HUQUKGRDKWB2WO4N3LKWP64FYS6YJGAQ, seg_pos={0 0}, upload=336945]: Long tail cancellation due to optimal threshold Oct 9 12:28:54.531648006 (382h57m39.993740489s) [noise=true , address=145.79.227.18:4006 , piece_id=FMUIMEPSKKC6C5RWKN6YLNLYOUDBCRS2CIXGHVA6MZWCZXFW6JMA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Oct 9 12:28:54.531656242 (382h57m39.993748725s) [noise=true , address=145.79.227.18:4006 , piece_id=FMUIMEPSKKC6C5RWKN6YLNLYOUDBCRS2CIXGHVA6MZWCZXFW6JMA, seg_pos={0 0}, upload=336945]: Long tail cancellation due to optimal threshold Oct 9 12:28:54.531725233 (382h57m39.993817716s) [noise=true , address=145.79.227.3:4150 , piece_id=4US3ZYUPQNOT4CHJMUUTCFIEIECELNIIRDDVLO4Z7NTN7L7QE3EQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 12:28:54.531736574 (382h57m39.993829077s) [noise=true , address=145.79.227.3:4150 , piece_id=4US3ZYUPQNOT4CHJMUUTCFIEIECELNIIRDDVLO4Z7NTN7L7QE3EQ, seg_pos={0 0}, upload=336945]: Long tail cancellation due to optimal threshold Oct 9 12:28:54.531762103 (382h57m39.993854596s) [seg_pos={0 0}, upload=336945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:28:54.531856422 (382h57m39.993948895s) [upload=336945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:28:54.531862905 (382h57m39.993955378s) [upload=336945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:28:54.958860302 (382h57m40.420952775s) [upload=336945]: Done uploading segments... Oct 9 12:28:54.958879008 (382h57m40.420971481s) [upload=336945]: Upload finished. err: <nil> Duration: 26.842842875s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X...X...X.] duration: 8.941656774s FAIL (2025-10-09 12:30:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 2E2ERXEHZ7ETMSL3JG26LDAK75SNEQD7PRGBILIUAQFJWTWYPGDA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: NLMPTFLKFV4CBW5MUGRNM65257PXLDOSNS23O34NSG7K6XV7X3XQ) context deadline exceeded ======LOGS====== Oct 9 12:28:51.645526499 (382h57m37.107618982s) [upload=336945]: Starting upload Oct 9 12:28:51.645626680 (382h57m37.107719173s) [upload=336945]: Done starting upload Oct 9 12:28:51.645903016 (382h57m37.107995499s) [upload=336945]: Uploading segments... Oct 9 12:28:51.646109018 (382h57m37.108201501s) [upload=336945]: Got next segment. Inline: false Oct 9 12:28:51.646190253 (382h57m37.108282736s) [upload=336945]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:28:51.646197367 (382h57m37.10828985s) [upload=336945]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:28:51.895659356 (382h57m37.357751839s) [seg_pos={0 0}, upload=336945]: Begin upload segment... Oct 9 12:28:51.895738718 (382h57m37.357831201s) [seg_pos={0 0}, upload=336945]: Done begin upload segment. Oct 9 12:28:51.895771059 (382h57m37.357863542s) [noise=true , address=45.91.116.5:4299 , piece_id=2E2ERXEHZ7ETMSL3JG26LDAK75SNEQD7PRGBILIUAQFJWTWYPGDA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.896756075 (382h57m37.358848558s) [noise=true , address=77.90.31.10:4519 , piece_id=P5BLQQF6JDOFB5RRICRA4BCHYABXWK7JKCODDONPO5KNY5LUBKMQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.897246589 (382h57m37.359339072s) [noise=true , address=45.91.116.14:4190 , piece_id=426KBG5VMF3X76DSOEGTM6NPGXG5BDNRXIVY2DRFK234ZBJPA5FA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.897703198 (382h57m37.359795682s) [noise=true , address=45.91.116.11:4118 , piece_id=NLMPTFLKFV4CBW5MUGRNM65257PXLDOSNS23O34NSG7K6XV7X3XQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.898150530 (382h57m37.360243014s) [noise=true , address=145.79.227.4:4185 , piece_id=WZGAAFLQ2UHAWQ7XJXKUATXAYMFJTIQ57OYZGBPZAXCOBFKW3WSQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.898609644 (382h57m37.360702128s) [noise=true , address=145.79.227.5:4222 , piece_id=3LPHHZJ7PIO3XC4QFER3HUQUKGRDKWB2WO4N3LKWP64FYS6YJGAQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.899094027 (382h57m37.36118651s) [noise=true , address=145.79.227.3:4150 , piece_id=4US3ZYUPQNOT4CHJMUUTCFIEIECELNIIRDDVLO4Z7NTN7L7QE3EQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.899566788 (382h57m37.361659271s) [noise=true , address=145.79.227.2:4114 , piece_id=RWVGXJNEZSGSK2W6BCHEMJL3QWTMYM3ZYRPPA4CZMNV5B2KJC2EQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.900128828 (382h57m37.362221311s) [noise=true , address=145.79.227.12:4767 , piece_id=75IDFDIKKU7S22O4D6JU3ILAOQVPN7GCAJMHOYLG4JD43M3RKW5A, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.900630332 (382h57m37.362722805s) [noise=true , address=145.79.227.15:4874 , piece_id=Z34KEBHVDIKNBGKZOQHNDLRWY6FIHDDP6DXG2RLN2WIBWL2MMJZA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.900960972 (382h57m37.363053455s) [noise=true , address=145.79.227.17:4969 , piece_id=Z7NELFATM54L6M7TN3QZY47F54Z5H76KMRJZCRELLSRM4G7M27WA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.901361484 (382h57m37.363453967s) [noise=true , address=145.79.227.18:4006 , piece_id=FMUIMEPSKKC6C5RWKN6YLNLYOUDBCRS2CIXGHVA6MZWCZXFW6JMA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.901680702 (382h57m37.363773185s) [noise=true , address=145.79.227.5:4223 , piece_id=EWICQ5IHQCEMXX4KQU6LADIB6Z54SSJ7JL4JNGBREMYQAGMULPPA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.905970862 (382h57m37.368063335s) [noise=true , address=145.79.227.11:4731 , piece_id=XHJB4F22MQ4RSAL54XIJYRH7M5JEKFKNIJ4SCPRVRVWMCXXPNBAQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.908040112 (382h57m37.370132585s) [noise=true , address=145.79.227.7:4295 , piece_id=EES54AYYAOIE6E67LRO3QBBOK63AANZ2U6RRHTFZBLNXVW2672JA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.910078714 (382h57m37.372171197s) [noise=true , address=45.91.116.8:4048 , piece_id=DU2ML2LKNW734TWWW3EXM6PLMI23NOGVCQR22V6AQ4YFNE67QVRA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:53.880123120 (382h57m39.342215603s) [upload=336945]: Next returned nil segment Oct 9 12:28:53.880146174 (382h57m39.342238657s) [upload=336945]: Waiting for error group managing segments... Oct 9 12:28:54.397570323 (382h57m39.859662796s) [noise=true , address=45.91.116.8:4048 , piece_id=DU2ML2LKNW734TWWW3EXM6PLMI23NOGVCQR22V6AQ4YFNE67QVRA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.427819273 (382h57m39.889911746s) [noise=true , address=145.79.227.7:4295 , piece_id=EES54AYYAOIE6E67LRO3QBBOK63AANZ2U6RRHTFZBLNXVW2672JA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.434534380 (382h57m39.896626873s) [noise=true , address=45.91.116.11:4118 , piece_id=NLMPTFLKFV4CBW5MUGRNM65257PXLDOSNS23O34NSG7K6XV7X3XQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.435744584 (382h57m39.897837068s) [noise=true , address=145.79.227.15:4874 , piece_id=Z34KEBHVDIKNBGKZOQHNDLRWY6FIHDDP6DXG2RLN2WIBWL2MMJZA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.436162019 (382h57m39.898254492s) [noise=true , address=145.79.227.11:4731 , piece_id=XHJB4F22MQ4RSAL54XIJYRH7M5JEKFKNIJ4SCPRVRVWMCXXPNBAQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.465006876 (382h57m39.927099359s) [noise=true , address=45.91.116.5:4299 , piece_id=2E2ERXEHZ7ETMSL3JG26LDAK75SNEQD7PRGBILIUAQFJWTWYPGDA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.478108986 (382h57m39.940201469s) [noise=true , address=45.91.116.14:4190 , piece_id=426KBG5VMF3X76DSOEGTM6NPGXG5BDNRXIVY2DRFK234ZBJPA5FA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.481253283 (382h57m39.943345756s) [noise=true , address=145.79.227.12:4767 , piece_id=75IDFDIKKU7S22O4D6JU3ILAOQVPN7GCAJMHOYLG4JD43M3RKW5A, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.485721652 (382h57m39.947814135s) [noise=true , address=145.79.227.17:4969 , piece_id=Z7NELFATM54L6M7TN3QZY47F54Z5H76KMRJZCRELLSRM4G7M27WA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.486622207 (382h57m39.94871468s) [noise=true , address=145.79.227.5:4223 , piece_id=EWICQ5IHQCEMXX4KQU6LADIB6Z54SSJ7JL4JNGBREMYQAGMULPPA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.493042041 (382h57m39.955134524s) [noise=true , address=145.79.227.4:4185 , piece_id=WZGAAFLQ2UHAWQ7XJXKUATXAYMFJTIQ57OYZGBPZAXCOBFKW3WSQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.493683773 (382h57m39.955776246s) [noise=true , address=145.79.227.2:4114 , piece_id=RWVGXJNEZSGSK2W6BCHEMJL3QWTMYM3ZYRPPA4CZMNV5B2KJC2EQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.531421124 (382h57m39.993513607s) [noise=true , address=77.90.31.10:4519 , piece_id=P5BLQQF6JDOFB5RRICRA4BCHYABXWK7JKCODDONPO5KNY5LUBKMQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.531455710 (382h57m39.993548193s) [seg_pos={0 0}, upload=336945]: Segment reached optimal threshold of 13 pieces. Oct 9 12:28:54.531610754 (382h57m39.993703237s) [noise=true , address=145.79.227.5:4222 , piece_id=3LPHHZJ7PIO3XC4QFER3HUQUKGRDKWB2WO4N3LKWP64FYS6YJGAQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Oct 9 12:28:54.531623670 (382h57m39.993716153s) [noise=true , address=145.79.227.5:4222 , piece_id=3LPHHZJ7PIO3XC4QFER3HUQUKGRDKWB2WO4N3LKWP64FYS6YJGAQ, seg_pos={0 0}, upload=336945]: Long tail cancellation due to optimal threshold Oct 9 12:28:54.531648006 (382h57m39.993740489s) [noise=true , address=145.79.227.18:4006 , piece_id=FMUIMEPSKKC6C5RWKN6YLNLYOUDBCRS2CIXGHVA6MZWCZXFW6JMA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Oct 9 12:28:54.531656242 (382h57m39.993748725s) [noise=true , address=145.79.227.18:4006 , piece_id=FMUIMEPSKKC6C5RWKN6YLNLYOUDBCRS2CIXGHVA6MZWCZXFW6JMA, seg_pos={0 0}, upload=336945]: Long tail cancellation due to optimal threshold Oct 9 12:28:54.531725233 (382h57m39.993817716s) [noise=true , address=145.79.227.3:4150 , piece_id=4US3ZYUPQNOT4CHJMUUTCFIEIECELNIIRDDVLO4Z7NTN7L7QE3EQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 12:28:54.531736574 (382h57m39.993829077s) [noise=true , address=145.79.227.3:4150 , piece_id=4US3ZYUPQNOT4CHJMUUTCFIEIECELNIIRDDVLO4Z7NTN7L7QE3EQ, seg_pos={0 0}, upload=336945]: Long tail cancellation due to optimal threshold Oct 9 12:28:54.531762103 (382h57m39.993854596s) [seg_pos={0 0}, upload=336945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:28:54.531856422 (382h57m39.993948895s) [upload=336945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:28:54.531862905 (382h57m39.993955378s) [upload=336945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:28:54.958860302 (382h57m40.420952775s) [upload=336945]: Done uploading segments... Oct 9 12:28:54.958879008 (382h57m40.420971481s) [upload=336945]: Upload finished. err: <nil> Duration: 8.941694805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X...X...X] duration: 30.002416119s FAIL (2025-10-09 12:29:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 2E2ERXEHZ7ETMSL3JG26LDAK75SNEQD7PRGBILIUAQFJWTWYPGDA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: NLMPTFLKFV4CBW5MUGRNM65257PXLDOSNS23O34NSG7K6XV7X3XQ) context deadline exceeded ======LOGS====== Oct 9 12:28:51.645526499 (382h57m37.107618982s) [upload=336945]: Starting upload Oct 9 12:28:51.645626680 (382h57m37.107719173s) [upload=336945]: Done starting upload Oct 9 12:28:51.645903016 (382h57m37.107995499s) [upload=336945]: Uploading segments... Oct 9 12:28:51.646109018 (382h57m37.108201501s) [upload=336945]: Got next segment. Inline: false Oct 9 12:28:51.646190253 (382h57m37.108282736s) [upload=336945]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:28:51.646197367 (382h57m37.10828985s) [upload=336945]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:28:51.895659356 (382h57m37.357751839s) [seg_pos={0 0}, upload=336945]: Begin upload segment... Oct 9 12:28:51.895738718 (382h57m37.357831201s) [seg_pos={0 0}, upload=336945]: Done begin upload segment. Oct 9 12:28:51.895771059 (382h57m37.357863542s) [noise=true , address=45.91.116.5:4299 , piece_id=2E2ERXEHZ7ETMSL3JG26LDAK75SNEQD7PRGBILIUAQFJWTWYPGDA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.896756075 (382h57m37.358848558s) [noise=true , address=77.90.31.10:4519 , piece_id=P5BLQQF6JDOFB5RRICRA4BCHYABXWK7JKCODDONPO5KNY5LUBKMQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.897246589 (382h57m37.359339072s) [noise=true , address=45.91.116.14:4190 , piece_id=426KBG5VMF3X76DSOEGTM6NPGXG5BDNRXIVY2DRFK234ZBJPA5FA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.897703198 (382h57m37.359795682s) [noise=true , address=45.91.116.11:4118 , piece_id=NLMPTFLKFV4CBW5MUGRNM65257PXLDOSNS23O34NSG7K6XV7X3XQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.898150530 (382h57m37.360243014s) [noise=true , address=145.79.227.4:4185 , piece_id=WZGAAFLQ2UHAWQ7XJXKUATXAYMFJTIQ57OYZGBPZAXCOBFKW3WSQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.898609644 (382h57m37.360702128s) [noise=true , address=145.79.227.5:4222 , piece_id=3LPHHZJ7PIO3XC4QFER3HUQUKGRDKWB2WO4N3LKWP64FYS6YJGAQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.899094027 (382h57m37.36118651s) [noise=true , address=145.79.227.3:4150 , piece_id=4US3ZYUPQNOT4CHJMUUTCFIEIECELNIIRDDVLO4Z7NTN7L7QE3EQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.899566788 (382h57m37.361659271s) [noise=true , address=145.79.227.2:4114 , piece_id=RWVGXJNEZSGSK2W6BCHEMJL3QWTMYM3ZYRPPA4CZMNV5B2KJC2EQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.900128828 (382h57m37.362221311s) [noise=true , address=145.79.227.12:4767 , piece_id=75IDFDIKKU7S22O4D6JU3ILAOQVPN7GCAJMHOYLG4JD43M3RKW5A, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.900630332 (382h57m37.362722805s) [noise=true , address=145.79.227.15:4874 , piece_id=Z34KEBHVDIKNBGKZOQHNDLRWY6FIHDDP6DXG2RLN2WIBWL2MMJZA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.900960972 (382h57m37.363053455s) [noise=true , address=145.79.227.17:4969 , piece_id=Z7NELFATM54L6M7TN3QZY47F54Z5H76KMRJZCRELLSRM4G7M27WA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.901361484 (382h57m37.363453967s) [noise=true , address=145.79.227.18:4006 , piece_id=FMUIMEPSKKC6C5RWKN6YLNLYOUDBCRS2CIXGHVA6MZWCZXFW6JMA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.901680702 (382h57m37.363773185s) [noise=true , address=145.79.227.5:4223 , piece_id=EWICQ5IHQCEMXX4KQU6LADIB6Z54SSJ7JL4JNGBREMYQAGMULPPA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.905970862 (382h57m37.368063335s) [noise=true , address=145.79.227.11:4731 , piece_id=XHJB4F22MQ4RSAL54XIJYRH7M5JEKFKNIJ4SCPRVRVWMCXXPNBAQ, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.908040112 (382h57m37.370132585s) [noise=true , address=145.79.227.7:4295 , piece_id=EES54AYYAOIE6E67LRO3QBBOK63AANZ2U6RRHTFZBLNXVW2672JA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:51.910078714 (382h57m37.372171197s) [noise=true , address=45.91.116.8:4048 , piece_id=DU2ML2LKNW734TWWW3EXM6PLMI23NOGVCQR22V6AQ4YFNE67QVRA, seg_pos={0 0}, upload=336945]: Uploading piece... Oct 9 12:28:53.880123120 (382h57m39.342215603s) [upload=336945]: Next returned nil segment Oct 9 12:28:53.880146174 (382h57m39.342238657s) [upload=336945]: Waiting for error group managing segments... Oct 9 12:28:54.397570323 (382h57m39.859662796s) [noise=true , address=45.91.116.8:4048 , piece_id=DU2ML2LKNW734TWWW3EXM6PLMI23NOGVCQR22V6AQ4YFNE67QVRA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.427819273 (382h57m39.889911746s) [noise=true , address=145.79.227.7:4295 , piece_id=EES54AYYAOIE6E67LRO3QBBOK63AANZ2U6RRHTFZBLNXVW2672JA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.434534380 (382h57m39.896626873s) [noise=true , address=45.91.116.11:4118 , piece_id=NLMPTFLKFV4CBW5MUGRNM65257PXLDOSNS23O34NSG7K6XV7X3XQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.435744584 (382h57m39.897837068s) [noise=true , address=145.79.227.15:4874 , piece_id=Z34KEBHVDIKNBGKZOQHNDLRWY6FIHDDP6DXG2RLN2WIBWL2MMJZA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.436162019 (382h57m39.898254492s) [noise=true , address=145.79.227.11:4731 , piece_id=XHJB4F22MQ4RSAL54XIJYRH7M5JEKFKNIJ4SCPRVRVWMCXXPNBAQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.465006876 (382h57m39.927099359s) [noise=true , address=45.91.116.5:4299 , piece_id=2E2ERXEHZ7ETMSL3JG26LDAK75SNEQD7PRGBILIUAQFJWTWYPGDA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.478108986 (382h57m39.940201469s) [noise=true , address=45.91.116.14:4190 , piece_id=426KBG5VMF3X76DSOEGTM6NPGXG5BDNRXIVY2DRFK234ZBJPA5FA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.481253283 (382h57m39.943345756s) [noise=true , address=145.79.227.12:4767 , piece_id=75IDFDIKKU7S22O4D6JU3ILAOQVPN7GCAJMHOYLG4JD43M3RKW5A, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.485721652 (382h57m39.947814135s) [noise=true , address=145.79.227.17:4969 , piece_id=Z7NELFATM54L6M7TN3QZY47F54Z5H76KMRJZCRELLSRM4G7M27WA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.486622207 (382h57m39.94871468s) [noise=true , address=145.79.227.5:4223 , piece_id=EWICQ5IHQCEMXX4KQU6LADIB6Z54SSJ7JL4JNGBREMYQAGMULPPA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.493042041 (382h57m39.955134524s) [noise=true , address=145.79.227.4:4185 , piece_id=WZGAAFLQ2UHAWQ7XJXKUATXAYMFJTIQ57OYZGBPZAXCOBFKW3WSQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.493683773 (382h57m39.955776246s) [noise=true , address=145.79.227.2:4114 , piece_id=RWVGXJNEZSGSK2W6BCHEMJL3QWTMYM3ZYRPPA4CZMNV5B2KJC2EQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.531421124 (382h57m39.993513607s) [noise=true , address=77.90.31.10:4519 , piece_id=P5BLQQF6JDOFB5RRICRA4BCHYABXWK7JKCODDONPO5KNY5LUBKMQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: <nil> Oct 9 12:28:54.531455710 (382h57m39.993548193s) [seg_pos={0 0}, upload=336945]: Segment reached optimal threshold of 13 pieces. Oct 9 12:28:54.531610754 (382h57m39.993703237s) [noise=true , address=145.79.227.5:4222 , piece_id=3LPHHZJ7PIO3XC4QFER3HUQUKGRDKWB2WO4N3LKWP64FYS6YJGAQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Oct 9 12:28:54.531623670 (382h57m39.993716153s) [noise=true , address=145.79.227.5:4222 , piece_id=3LPHHZJ7PIO3XC4QFER3HUQUKGRDKWB2WO4N3LKWP64FYS6YJGAQ, seg_pos={0 0}, upload=336945]: Long tail cancellation due to optimal threshold Oct 9 12:28:54.531648006 (382h57m39.993740489s) [noise=true , address=145.79.227.18:4006 , piece_id=FMUIMEPSKKC6C5RWKN6YLNLYOUDBCRS2CIXGHVA6MZWCZXFW6JMA, seg_pos={0 0}, upload=336945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: expected piece hash; context canceled; EOF Oct 9 12:28:54.531656242 (382h57m39.993748725s) [noise=true , address=145.79.227.18:4006 , piece_id=FMUIMEPSKKC6C5RWKN6YLNLYOUDBCRS2CIXGHVA6MZWCZXFW6JMA, seg_pos={0 0}, upload=336945]: Long tail cancellation due to optimal threshold Oct 9 12:28:54.531725233 (382h57m39.993817716s) [noise=true , address=145.79.227.3:4150 , piece_id=4US3ZYUPQNOT4CHJMUUTCFIEIECELNIIRDDVLO4Z7NTN7L7QE3EQ, seg_pos={0 0}, upload=336945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 12:28:54.531736574 (382h57m39.993829077s) [noise=true , address=145.79.227.3:4150 , piece_id=4US3ZYUPQNOT4CHJMUUTCFIEIECELNIIRDDVLO4Z7NTN7L7QE3EQ, seg_pos={0 0}, upload=336945]: Long tail cancellation due to optimal threshold Oct 9 12:28:54.531762103 (382h57m39.993854596s) [seg_pos={0 0}, upload=336945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:28:54.531856422 (382h57m39.993948895s) [upload=336945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:28:54.531862905 (382h57m39.993955378s) [upload=336945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:28:54.958860302 (382h57m40.420952775s) [upload=336945]: Done uploading segments... Oct 9 12:28:54.958879008 (382h57m40.420971481s) [upload=336945]: Upload finished. err: <nil> Duration: 30.002460714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X...X...] duration: 8.012676325s FAIL (2025-10-09 12:28:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: FOWPFEKBM3YUNYZELHYY2EIEMN65MJN3A6XYQRD3CDIKORRQAIKA) context deadline exceeded; piecestore: (Node ID: 12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, Piece ID: BLHAGWVLRELTP32R7SMCXOLGMWAJHGWXZ6XVUYQGDJ2BKZ55DEZQ) context deadline exceeded ======LOGS====== Oct 9 12:26:40.842434521 (382h55m26.304526994s) [upload=336908]: Starting upload Oct 9 12:26:40.842510034 (382h55m26.304602517s) [upload=336908]: Done starting upload Oct 9 12:26:40.842764820 (382h55m26.304857303s) [upload=336908]: Uploading segments... Oct 9 12:26:40.842845704 (382h55m26.304938187s) [upload=336908]: Got next segment. Inline: false Oct 9 12:26:40.846390183 (382h55m26.308482666s) [upload=336908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:26:40.846408758 (382h55m26.308501231s) [upload=336908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:26:41.094330866 (382h55m26.556423379s) [seg_pos={0 0}, upload=336908]: Begin upload segment... Oct 9 12:26:41.094403173 (382h55m26.556495646s) [seg_pos={0 0}, upload=336908]: Done begin upload segment. Oct 9 12:26:41.094419845 (382h55m26.556512329s) [noise=true , address=45.91.116.6:4974 , piece_id=V76LUWFCQP7T74HFEXHBMJXQZRSQSFOAX5A3GZT3MBQSIIUWONWA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.094439152 (382h55m26.556531635s) [noise=true , address=145.79.227.15:4876 , piece_id=XLUM526JLSMNMNDRDKBXG36B4G4KVL6WLQIJHVA5M4FWUMKXEEYQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.094889068 (382h55m26.556981552s) [noise=true , address=145.79.227.2:4115 , piece_id=X5QIBLDXTB6WNI5NM3TGZJO6CJQZLGGDCS6BEMTYARSCAOL6C5UQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.095193689 (382h55m26.557286172s) [noise=true , address=145.79.227.19:4078 , piece_id=F6OECBUFUU2PWSYMROTSJY47TCTHS2NJHID6IYOQHGMU3IGV5RXA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.095512736 (382h55m26.557605209s) [noise=true , address=145.79.227.12:4767 , piece_id=FK2KKBEU7VNJZVOWWQ2US5XVT5KJIVTOWG7POKPCJWQ5IQZZ6BFQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.095836142 (382h55m26.557928615s) [noise=true , address=145.79.227.18:4005 , piece_id=EPBY2RIJYADSKDSA7YRIDIRORFEH3TFNWSJU33PZL2P6U2HAI3HQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.096156573 (382h55m26.558249046s) [noise=true , address=145.79.227.11:4731 , piece_id=NZRQU3JCK46QBBOT6F5MSUYPF3LMWZ6LXSSVUSAOIRFACKJNEBPQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.096661644 (382h55m26.558754117s) [noise=true , address=45.91.116.9:4262 , piece_id=NPYFDG67EYKWI25Z5KC6RV4HA3SWK5HTHVFDKRLP3AL43AUPXA7Q, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.097164993 (382h55m26.559257476s) [noise=true , address=145.79.227.3:4150 , piece_id=3P7C3WB4WNGHZQYIVWNHJHOMQFBHAVHY7PNNUQWOFG7GFEVSTPGA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.097620761 (382h55m26.559713244s) [noise=true , address=45.91.116.7:4011 , piece_id=FOWPFEKBM3YUNYZELHYY2EIEMN65MJN3A6XYQRD3CDIKORRQAIKA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.098718380 (382h55m26.560810863s) [noise=true , address=145.79.227.16:4909 , piece_id=ECE7RMOL4WFHTM2ETWD46FCGO4DSMGOKMLN4RD3LMF5E43AC3TIQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.099245084 (382h55m26.561337567s) [noise=true , address=77.90.31.17:4770 , piece_id=MSJ3XZFBOOPR33OXQN7FSBFTHXE5DPCYWRGMZA7YCUT2IIYVZ4OQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.099941009 (382h55m26.562033502s) [noise=true , address=145.79.227.17:4972 , piece_id=ITGSR44OGJUPEZLXNDRVNARQ57SZVBA3DSKG6VCAACLDYMDT6ZGQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.100621013 (382h55m26.562713497s) [noise=true , address=145.79.227.14:4840 , piece_id=QNUM5BN6X2CZF5G7OYWXC5X2OTH374A3OEWJTBU3NSBXG5PFRNKQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.102221601 (382h55m26.564314084s) [noise=true , address=145.79.227.2:4114 , piece_id=SMDLDOYCKUJNEXS3N7JUJECZEA74PJXXDPB2TAGWFSS5HJ4Y5V6A, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.104416040 (382h55m26.566508523s) [noise=true , address=77.90.31.16:4733 , piece_id=BLHAGWVLRELTP32R7SMCXOLGMWAJHGWXZ6XVUYQGDJ2BKZ55DEZQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:42.944427443 (382h55m28.406519927s) [upload=336908]: Next returned nil segment Oct 9 12:26:42.944609239 (382h55m28.406701712s) [upload=336908]: Waiting for error group managing segments... Oct 9 12:26:43.308230354 (382h55m28.770322837s) [noise=true , address=45.91.116.6:4974 , piece_id=V76LUWFCQP7T74HFEXHBMJXQZRSQSFOAX5A3GZT3MBQSIIUWONWA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.310855472 (382h55m28.772947955s) [noise=true , address=77.90.31.16:4733 , piece_id=BLHAGWVLRELTP32R7SMCXOLGMWAJHGWXZ6XVUYQGDJ2BKZ55DEZQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.376033476 (382h55m28.838125959s) [noise=true , address=45.91.116.7:4011 , piece_id=FOWPFEKBM3YUNYZELHYY2EIEMN65MJN3A6XYQRD3CDIKORRQAIKA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.550644454 (382h55m29.012736937s) [noise=true , address=45.91.116.9:4262 , piece_id=NPYFDG67EYKWI25Z5KC6RV4HA3SWK5HTHVFDKRLP3AL43AUPXA7Q, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.600245505 (382h55m29.062337978s) [noise=true , address=77.90.31.17:4770 , piece_id=MSJ3XZFBOOPR33OXQN7FSBFTHXE5DPCYWRGMZA7YCUT2IIYVZ4OQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.731440649 (382h55m29.193533132s) [noise=true , address=145.79.227.12:4767 , piece_id=FK2KKBEU7VNJZVOWWQ2US5XVT5KJIVTOWG7POKPCJWQ5IQZZ6BFQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.798391097 (382h55m29.26048358s) [noise=true , address=145.79.227.19:4078 , piece_id=F6OECBUFUU2PWSYMROTSJY47TCTHS2NJHID6IYOQHGMU3IGV5RXA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.812053303 (382h55m29.274145786s) [noise=true , address=145.79.227.18:4005 , piece_id=EPBY2RIJYADSKDSA7YRIDIRORFEH3TFNWSJU33PZL2P6U2HAI3HQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.817079344 (382h55m29.279171837s) [noise=true , address=145.79.227.17:4972 , piece_id=ITGSR44OGJUPEZLXNDRVNARQ57SZVBA3DSKG6VCAACLDYMDT6ZGQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.836708482 (382h55m29.298800965s) [noise=true , address=145.79.227.2:4115 , piece_id=X5QIBLDXTB6WNI5NM3TGZJO6CJQZLGGDCS6BEMTYARSCAOL6C5UQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.918643605 (382h55m29.380736088s) [noise=true , address=145.79.227.3:4150 , piece_id=3P7C3WB4WNGHZQYIVWNHJHOMQFBHAVHY7PNNUQWOFG7GFEVSTPGA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.922959764 (382h55m29.385052247s) [noise=true , address=145.79.227.11:4731 , piece_id=NZRQU3JCK46QBBOT6F5MSUYPF3LMWZ6LXSSVUSAOIRFACKJNEBPQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.927920290 (382h55m29.390012773s) [noise=true , address=145.79.227.14:4840 , piece_id=QNUM5BN6X2CZF5G7OYWXC5X2OTH374A3OEWJTBU3NSBXG5PFRNKQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.927944315 (382h55m29.390036788s) [seg_pos={0 0}, upload=336908]: Segment reached optimal threshold of 13 pieces. Oct 9 12:26:43.928058163 (382h55m29.390150646s) [noise=true , address=145.79.227.2:4114 , piece_id=SMDLDOYCKUJNEXS3N7JUJECZEA74PJXXDPB2TAGWFSS5HJ4Y5V6A, seg_pos={0 0}, upload=336908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: expected piece hash; context canceled; EOF Oct 9 12:26:43.928066528 (382h55m29.390159011s) [noise=true , address=145.79.227.2:4114 , piece_id=SMDLDOYCKUJNEXS3N7JUJECZEA74PJXXDPB2TAGWFSS5HJ4Y5V6A, seg_pos={0 0}, upload=336908]: Long tail cancellation due to optimal threshold Oct 9 12:26:43.928176477 (382h55m29.39026895s) [noise=true , address=145.79.227.15:4876 , piece_id=XLUM526JLSMNMNDRDKBXG36B4G4KVL6WLQIJHVA5M4FWUMKXEEYQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Oct 9 12:26:43.928184553 (382h55m29.390277026s) [noise=true , address=145.79.227.15:4876 , piece_id=XLUM526JLSMNMNDRDKBXG36B4G4KVL6WLQIJHVA5M4FWUMKXEEYQ, seg_pos={0 0}, upload=336908]: Long tail cancellation due to optimal threshold Oct 9 12:26:43.928292378 (382h55m29.390384871s) [noise=true , address=145.79.227.16:4909 , piece_id=ECE7RMOL4WFHTM2ETWD46FCGO4DSMGOKMLN4RD3LMF5E43AC3TIQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Oct 9 12:26:43.928300202 (382h55m29.390392686s) [noise=true , address=145.79.227.16:4909 , piece_id=ECE7RMOL4WFHTM2ETWD46FCGO4DSMGOKMLN4RD3LMF5E43AC3TIQ, seg_pos={0 0}, upload=336908]: Long tail cancellation due to optimal threshold Oct 9 12:26:43.928319760 (382h55m29.390412243s) [seg_pos={0 0}, upload=336908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:26:43.928402547 (382h55m29.39049503s) [upload=336908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:26:43.928410083 (382h55m29.390502566s) [upload=336908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:26:44.244007830 (382h55m29.706100333s) [upload=336908]: Done uploading segments... Oct 9 12:26:44.244032437 (382h55m29.70612492s) [upload=336908]: Upload finished. err: <nil> Duration: 8.012708808s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 26.032997777s FAIL (2025-10-09 12:28:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: FOWPFEKBM3YUNYZELHYY2EIEMN65MJN3A6XYQRD3CDIKORRQAIKA) context deadline exceeded; piecestore: (Node ID: 12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, Piece ID: BLHAGWVLRELTP32R7SMCXOLGMWAJHGWXZ6XVUYQGDJ2BKZ55DEZQ) context deadline exceeded ======LOGS====== Oct 9 12:26:40.842434521 (382h55m26.304526994s) [upload=336908]: Starting upload Oct 9 12:26:40.842510034 (382h55m26.304602517s) [upload=336908]: Done starting upload Oct 9 12:26:40.842764820 (382h55m26.304857303s) [upload=336908]: Uploading segments... Oct 9 12:26:40.842845704 (382h55m26.304938187s) [upload=336908]: Got next segment. Inline: false Oct 9 12:26:40.846390183 (382h55m26.308482666s) [upload=336908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:26:40.846408758 (382h55m26.308501231s) [upload=336908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:26:41.094330866 (382h55m26.556423379s) [seg_pos={0 0}, upload=336908]: Begin upload segment... Oct 9 12:26:41.094403173 (382h55m26.556495646s) [seg_pos={0 0}, upload=336908]: Done begin upload segment. Oct 9 12:26:41.094419845 (382h55m26.556512329s) [noise=true , address=45.91.116.6:4974 , piece_id=V76LUWFCQP7T74HFEXHBMJXQZRSQSFOAX5A3GZT3MBQSIIUWONWA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.094439152 (382h55m26.556531635s) [noise=true , address=145.79.227.15:4876 , piece_id=XLUM526JLSMNMNDRDKBXG36B4G4KVL6WLQIJHVA5M4FWUMKXEEYQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.094889068 (382h55m26.556981552s) [noise=true , address=145.79.227.2:4115 , piece_id=X5QIBLDXTB6WNI5NM3TGZJO6CJQZLGGDCS6BEMTYARSCAOL6C5UQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.095193689 (382h55m26.557286172s) [noise=true , address=145.79.227.19:4078 , piece_id=F6OECBUFUU2PWSYMROTSJY47TCTHS2NJHID6IYOQHGMU3IGV5RXA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.095512736 (382h55m26.557605209s) [noise=true , address=145.79.227.12:4767 , piece_id=FK2KKBEU7VNJZVOWWQ2US5XVT5KJIVTOWG7POKPCJWQ5IQZZ6BFQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.095836142 (382h55m26.557928615s) [noise=true , address=145.79.227.18:4005 , piece_id=EPBY2RIJYADSKDSA7YRIDIRORFEH3TFNWSJU33PZL2P6U2HAI3HQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.096156573 (382h55m26.558249046s) [noise=true , address=145.79.227.11:4731 , piece_id=NZRQU3JCK46QBBOT6F5MSUYPF3LMWZ6LXSSVUSAOIRFACKJNEBPQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.096661644 (382h55m26.558754117s) [noise=true , address=45.91.116.9:4262 , piece_id=NPYFDG67EYKWI25Z5KC6RV4HA3SWK5HTHVFDKRLP3AL43AUPXA7Q, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.097164993 (382h55m26.559257476s) [noise=true , address=145.79.227.3:4150 , piece_id=3P7C3WB4WNGHZQYIVWNHJHOMQFBHAVHY7PNNUQWOFG7GFEVSTPGA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.097620761 (382h55m26.559713244s) [noise=true , address=45.91.116.7:4011 , piece_id=FOWPFEKBM3YUNYZELHYY2EIEMN65MJN3A6XYQRD3CDIKORRQAIKA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.098718380 (382h55m26.560810863s) [noise=true , address=145.79.227.16:4909 , piece_id=ECE7RMOL4WFHTM2ETWD46FCGO4DSMGOKMLN4RD3LMF5E43AC3TIQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.099245084 (382h55m26.561337567s) [noise=true , address=77.90.31.17:4770 , piece_id=MSJ3XZFBOOPR33OXQN7FSBFTHXE5DPCYWRGMZA7YCUT2IIYVZ4OQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.099941009 (382h55m26.562033502s) [noise=true , address=145.79.227.17:4972 , piece_id=ITGSR44OGJUPEZLXNDRVNARQ57SZVBA3DSKG6VCAACLDYMDT6ZGQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.100621013 (382h55m26.562713497s) [noise=true , address=145.79.227.14:4840 , piece_id=QNUM5BN6X2CZF5G7OYWXC5X2OTH374A3OEWJTBU3NSBXG5PFRNKQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.102221601 (382h55m26.564314084s) [noise=true , address=145.79.227.2:4114 , piece_id=SMDLDOYCKUJNEXS3N7JUJECZEA74PJXXDPB2TAGWFSS5HJ4Y5V6A, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.104416040 (382h55m26.566508523s) [noise=true , address=77.90.31.16:4733 , piece_id=BLHAGWVLRELTP32R7SMCXOLGMWAJHGWXZ6XVUYQGDJ2BKZ55DEZQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:42.944427443 (382h55m28.406519927s) [upload=336908]: Next returned nil segment Oct 9 12:26:42.944609239 (382h55m28.406701712s) [upload=336908]: Waiting for error group managing segments... Oct 9 12:26:43.308230354 (382h55m28.770322837s) [noise=true , address=45.91.116.6:4974 , piece_id=V76LUWFCQP7T74HFEXHBMJXQZRSQSFOAX5A3GZT3MBQSIIUWONWA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.310855472 (382h55m28.772947955s) [noise=true , address=77.90.31.16:4733 , piece_id=BLHAGWVLRELTP32R7SMCXOLGMWAJHGWXZ6XVUYQGDJ2BKZ55DEZQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.376033476 (382h55m28.838125959s) [noise=true , address=45.91.116.7:4011 , piece_id=FOWPFEKBM3YUNYZELHYY2EIEMN65MJN3A6XYQRD3CDIKORRQAIKA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.550644454 (382h55m29.012736937s) [noise=true , address=45.91.116.9:4262 , piece_id=NPYFDG67EYKWI25Z5KC6RV4HA3SWK5HTHVFDKRLP3AL43AUPXA7Q, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.600245505 (382h55m29.062337978s) [noise=true , address=77.90.31.17:4770 , piece_id=MSJ3XZFBOOPR33OXQN7FSBFTHXE5DPCYWRGMZA7YCUT2IIYVZ4OQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.731440649 (382h55m29.193533132s) [noise=true , address=145.79.227.12:4767 , piece_id=FK2KKBEU7VNJZVOWWQ2US5XVT5KJIVTOWG7POKPCJWQ5IQZZ6BFQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.798391097 (382h55m29.26048358s) [noise=true , address=145.79.227.19:4078 , piece_id=F6OECBUFUU2PWSYMROTSJY47TCTHS2NJHID6IYOQHGMU3IGV5RXA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.812053303 (382h55m29.274145786s) [noise=true , address=145.79.227.18:4005 , piece_id=EPBY2RIJYADSKDSA7YRIDIRORFEH3TFNWSJU33PZL2P6U2HAI3HQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.817079344 (382h55m29.279171837s) [noise=true , address=145.79.227.17:4972 , piece_id=ITGSR44OGJUPEZLXNDRVNARQ57SZVBA3DSKG6VCAACLDYMDT6ZGQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.836708482 (382h55m29.298800965s) [noise=true , address=145.79.227.2:4115 , piece_id=X5QIBLDXTB6WNI5NM3TGZJO6CJQZLGGDCS6BEMTYARSCAOL6C5UQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.918643605 (382h55m29.380736088s) [noise=true , address=145.79.227.3:4150 , piece_id=3P7C3WB4WNGHZQYIVWNHJHOMQFBHAVHY7PNNUQWOFG7GFEVSTPGA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.922959764 (382h55m29.385052247s) [noise=true , address=145.79.227.11:4731 , piece_id=NZRQU3JCK46QBBOT6F5MSUYPF3LMWZ6LXSSVUSAOIRFACKJNEBPQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.927920290 (382h55m29.390012773s) [noise=true , address=145.79.227.14:4840 , piece_id=QNUM5BN6X2CZF5G7OYWXC5X2OTH374A3OEWJTBU3NSBXG5PFRNKQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.927944315 (382h55m29.390036788s) [seg_pos={0 0}, upload=336908]: Segment reached optimal threshold of 13 pieces. Oct 9 12:26:43.928058163 (382h55m29.390150646s) [noise=true , address=145.79.227.2:4114 , piece_id=SMDLDOYCKUJNEXS3N7JUJECZEA74PJXXDPB2TAGWFSS5HJ4Y5V6A, seg_pos={0 0}, upload=336908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: expected piece hash; context canceled; EOF Oct 9 12:26:43.928066528 (382h55m29.390159011s) [noise=true , address=145.79.227.2:4114 , piece_id=SMDLDOYCKUJNEXS3N7JUJECZEA74PJXXDPB2TAGWFSS5HJ4Y5V6A, seg_pos={0 0}, upload=336908]: Long tail cancellation due to optimal threshold Oct 9 12:26:43.928176477 (382h55m29.39026895s) [noise=true , address=145.79.227.15:4876 , piece_id=XLUM526JLSMNMNDRDKBXG36B4G4KVL6WLQIJHVA5M4FWUMKXEEYQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Oct 9 12:26:43.928184553 (382h55m29.390277026s) [noise=true , address=145.79.227.15:4876 , piece_id=XLUM526JLSMNMNDRDKBXG36B4G4KVL6WLQIJHVA5M4FWUMKXEEYQ, seg_pos={0 0}, upload=336908]: Long tail cancellation due to optimal threshold Oct 9 12:26:43.928292378 (382h55m29.390384871s) [noise=true , address=145.79.227.16:4909 , piece_id=ECE7RMOL4WFHTM2ETWD46FCGO4DSMGOKMLN4RD3LMF5E43AC3TIQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Oct 9 12:26:43.928300202 (382h55m29.390392686s) [noise=true , address=145.79.227.16:4909 , piece_id=ECE7RMOL4WFHTM2ETWD46FCGO4DSMGOKMLN4RD3LMF5E43AC3TIQ, seg_pos={0 0}, upload=336908]: Long tail cancellation due to optimal threshold Oct 9 12:26:43.928319760 (382h55m29.390412243s) [seg_pos={0 0}, upload=336908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:26:43.928402547 (382h55m29.39049503s) [upload=336908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:26:43.928410083 (382h55m29.390502566s) [upload=336908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:26:44.244007830 (382h55m29.706100333s) [upload=336908]: Done uploading segments... Oct 9 12:26:44.244032437 (382h55m29.70612492s) [upload=336908]: Upload finished. err: <nil> Duration: 26.033028936s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 29.783090936s FAIL (2025-10-09 12:27:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: FOWPFEKBM3YUNYZELHYY2EIEMN65MJN3A6XYQRD3CDIKORRQAIKA) context deadline exceeded; piecestore: (Node ID: 12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, Piece ID: BLHAGWVLRELTP32R7SMCXOLGMWAJHGWXZ6XVUYQGDJ2BKZ55DEZQ) context deadline exceeded ======LOGS====== Oct 9 12:26:40.842434521 (382h55m26.304526994s) [upload=336908]: Starting upload Oct 9 12:26:40.842510034 (382h55m26.304602517s) [upload=336908]: Done starting upload Oct 9 12:26:40.842764820 (382h55m26.304857303s) [upload=336908]: Uploading segments... Oct 9 12:26:40.842845704 (382h55m26.304938187s) [upload=336908]: Got next segment. Inline: false Oct 9 12:26:40.846390183 (382h55m26.308482666s) [upload=336908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:26:40.846408758 (382h55m26.308501231s) [upload=336908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:26:41.094330866 (382h55m26.556423379s) [seg_pos={0 0}, upload=336908]: Begin upload segment... Oct 9 12:26:41.094403173 (382h55m26.556495646s) [seg_pos={0 0}, upload=336908]: Done begin upload segment. Oct 9 12:26:41.094419845 (382h55m26.556512329s) [noise=true , address=45.91.116.6:4974 , piece_id=V76LUWFCQP7T74HFEXHBMJXQZRSQSFOAX5A3GZT3MBQSIIUWONWA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.094439152 (382h55m26.556531635s) [noise=true , address=145.79.227.15:4876 , piece_id=XLUM526JLSMNMNDRDKBXG36B4G4KVL6WLQIJHVA5M4FWUMKXEEYQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.094889068 (382h55m26.556981552s) [noise=true , address=145.79.227.2:4115 , piece_id=X5QIBLDXTB6WNI5NM3TGZJO6CJQZLGGDCS6BEMTYARSCAOL6C5UQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.095193689 (382h55m26.557286172s) [noise=true , address=145.79.227.19:4078 , piece_id=F6OECBUFUU2PWSYMROTSJY47TCTHS2NJHID6IYOQHGMU3IGV5RXA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.095512736 (382h55m26.557605209s) [noise=true , address=145.79.227.12:4767 , piece_id=FK2KKBEU7VNJZVOWWQ2US5XVT5KJIVTOWG7POKPCJWQ5IQZZ6BFQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.095836142 (382h55m26.557928615s) [noise=true , address=145.79.227.18:4005 , piece_id=EPBY2RIJYADSKDSA7YRIDIRORFEH3TFNWSJU33PZL2P6U2HAI3HQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.096156573 (382h55m26.558249046s) [noise=true , address=145.79.227.11:4731 , piece_id=NZRQU3JCK46QBBOT6F5MSUYPF3LMWZ6LXSSVUSAOIRFACKJNEBPQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.096661644 (382h55m26.558754117s) [noise=true , address=45.91.116.9:4262 , piece_id=NPYFDG67EYKWI25Z5KC6RV4HA3SWK5HTHVFDKRLP3AL43AUPXA7Q, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.097164993 (382h55m26.559257476s) [noise=true , address=145.79.227.3:4150 , piece_id=3P7C3WB4WNGHZQYIVWNHJHOMQFBHAVHY7PNNUQWOFG7GFEVSTPGA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.097620761 (382h55m26.559713244s) [noise=true , address=45.91.116.7:4011 , piece_id=FOWPFEKBM3YUNYZELHYY2EIEMN65MJN3A6XYQRD3CDIKORRQAIKA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.098718380 (382h55m26.560810863s) [noise=true , address=145.79.227.16:4909 , piece_id=ECE7RMOL4WFHTM2ETWD46FCGO4DSMGOKMLN4RD3LMF5E43AC3TIQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.099245084 (382h55m26.561337567s) [noise=true , address=77.90.31.17:4770 , piece_id=MSJ3XZFBOOPR33OXQN7FSBFTHXE5DPCYWRGMZA7YCUT2IIYVZ4OQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.099941009 (382h55m26.562033502s) [noise=true , address=145.79.227.17:4972 , piece_id=ITGSR44OGJUPEZLXNDRVNARQ57SZVBA3DSKG6VCAACLDYMDT6ZGQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.100621013 (382h55m26.562713497s) [noise=true , address=145.79.227.14:4840 , piece_id=QNUM5BN6X2CZF5G7OYWXC5X2OTH374A3OEWJTBU3NSBXG5PFRNKQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.102221601 (382h55m26.564314084s) [noise=true , address=145.79.227.2:4114 , piece_id=SMDLDOYCKUJNEXS3N7JUJECZEA74PJXXDPB2TAGWFSS5HJ4Y5V6A, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.104416040 (382h55m26.566508523s) [noise=true , address=77.90.31.16:4733 , piece_id=BLHAGWVLRELTP32R7SMCXOLGMWAJHGWXZ6XVUYQGDJ2BKZ55DEZQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:42.944427443 (382h55m28.406519927s) [upload=336908]: Next returned nil segment Oct 9 12:26:42.944609239 (382h55m28.406701712s) [upload=336908]: Waiting for error group managing segments... Oct 9 12:26:43.308230354 (382h55m28.770322837s) [noise=true , address=45.91.116.6:4974 , piece_id=V76LUWFCQP7T74HFEXHBMJXQZRSQSFOAX5A3GZT3MBQSIIUWONWA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.310855472 (382h55m28.772947955s) [noise=true , address=77.90.31.16:4733 , piece_id=BLHAGWVLRELTP32R7SMCXOLGMWAJHGWXZ6XVUYQGDJ2BKZ55DEZQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.376033476 (382h55m28.838125959s) [noise=true , address=45.91.116.7:4011 , piece_id=FOWPFEKBM3YUNYZELHYY2EIEMN65MJN3A6XYQRD3CDIKORRQAIKA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.550644454 (382h55m29.012736937s) [noise=true , address=45.91.116.9:4262 , piece_id=NPYFDG67EYKWI25Z5KC6RV4HA3SWK5HTHVFDKRLP3AL43AUPXA7Q, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.600245505 (382h55m29.062337978s) [noise=true , address=77.90.31.17:4770 , piece_id=MSJ3XZFBOOPR33OXQN7FSBFTHXE5DPCYWRGMZA7YCUT2IIYVZ4OQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.731440649 (382h55m29.193533132s) [noise=true , address=145.79.227.12:4767 , piece_id=FK2KKBEU7VNJZVOWWQ2US5XVT5KJIVTOWG7POKPCJWQ5IQZZ6BFQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.798391097 (382h55m29.26048358s) [noise=true , address=145.79.227.19:4078 , piece_id=F6OECBUFUU2PWSYMROTSJY47TCTHS2NJHID6IYOQHGMU3IGV5RXA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.812053303 (382h55m29.274145786s) [noise=true , address=145.79.227.18:4005 , piece_id=EPBY2RIJYADSKDSA7YRIDIRORFEH3TFNWSJU33PZL2P6U2HAI3HQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.817079344 (382h55m29.279171837s) [noise=true , address=145.79.227.17:4972 , piece_id=ITGSR44OGJUPEZLXNDRVNARQ57SZVBA3DSKG6VCAACLDYMDT6ZGQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.836708482 (382h55m29.298800965s) [noise=true , address=145.79.227.2:4115 , piece_id=X5QIBLDXTB6WNI5NM3TGZJO6CJQZLGGDCS6BEMTYARSCAOL6C5UQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.918643605 (382h55m29.380736088s) [noise=true , address=145.79.227.3:4150 , piece_id=3P7C3WB4WNGHZQYIVWNHJHOMQFBHAVHY7PNNUQWOFG7GFEVSTPGA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.922959764 (382h55m29.385052247s) [noise=true , address=145.79.227.11:4731 , piece_id=NZRQU3JCK46QBBOT6F5MSUYPF3LMWZ6LXSSVUSAOIRFACKJNEBPQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.927920290 (382h55m29.390012773s) [noise=true , address=145.79.227.14:4840 , piece_id=QNUM5BN6X2CZF5G7OYWXC5X2OTH374A3OEWJTBU3NSBXG5PFRNKQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.927944315 (382h55m29.390036788s) [seg_pos={0 0}, upload=336908]: Segment reached optimal threshold of 13 pieces. Oct 9 12:26:43.928058163 (382h55m29.390150646s) [noise=true , address=145.79.227.2:4114 , piece_id=SMDLDOYCKUJNEXS3N7JUJECZEA74PJXXDPB2TAGWFSS5HJ4Y5V6A, seg_pos={0 0}, upload=336908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: expected piece hash; context canceled; EOF Oct 9 12:26:43.928066528 (382h55m29.390159011s) [noise=true , address=145.79.227.2:4114 , piece_id=SMDLDOYCKUJNEXS3N7JUJECZEA74PJXXDPB2TAGWFSS5HJ4Y5V6A, seg_pos={0 0}, upload=336908]: Long tail cancellation due to optimal threshold Oct 9 12:26:43.928176477 (382h55m29.39026895s) [noise=true , address=145.79.227.15:4876 , piece_id=XLUM526JLSMNMNDRDKBXG36B4G4KVL6WLQIJHVA5M4FWUMKXEEYQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Oct 9 12:26:43.928184553 (382h55m29.390277026s) [noise=true , address=145.79.227.15:4876 , piece_id=XLUM526JLSMNMNDRDKBXG36B4G4KVL6WLQIJHVA5M4FWUMKXEEYQ, seg_pos={0 0}, upload=336908]: Long tail cancellation due to optimal threshold Oct 9 12:26:43.928292378 (382h55m29.390384871s) [noise=true , address=145.79.227.16:4909 , piece_id=ECE7RMOL4WFHTM2ETWD46FCGO4DSMGOKMLN4RD3LMF5E43AC3TIQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Oct 9 12:26:43.928300202 (382h55m29.390392686s) [noise=true , address=145.79.227.16:4909 , piece_id=ECE7RMOL4WFHTM2ETWD46FCGO4DSMGOKMLN4RD3LMF5E43AC3TIQ, seg_pos={0 0}, upload=336908]: Long tail cancellation due to optimal threshold Oct 9 12:26:43.928319760 (382h55m29.390412243s) [seg_pos={0 0}, upload=336908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:26:43.928402547 (382h55m29.39049503s) [upload=336908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:26:43.928410083 (382h55m29.390502566s) [upload=336908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:26:44.244007830 (382h55m29.706100333s) [upload=336908]: Done uploading segments... Oct 9 12:26:44.244032437 (382h55m29.70612492s) [upload=336908]: Upload finished. err: <nil> Duration: 29.783120381s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 30.007170799s FAIL (2025-10-09 12:27:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: FOWPFEKBM3YUNYZELHYY2EIEMN65MJN3A6XYQRD3CDIKORRQAIKA) context deadline exceeded; piecestore: (Node ID: 12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, Piece ID: BLHAGWVLRELTP32R7SMCXOLGMWAJHGWXZ6XVUYQGDJ2BKZ55DEZQ) context deadline exceeded ======LOGS====== Oct 9 12:26:40.842434521 (382h55m26.304526994s) [upload=336908]: Starting upload Oct 9 12:26:40.842510034 (382h55m26.304602517s) [upload=336908]: Done starting upload Oct 9 12:26:40.842764820 (382h55m26.304857303s) [upload=336908]: Uploading segments... Oct 9 12:26:40.842845704 (382h55m26.304938187s) [upload=336908]: Got next segment. Inline: false Oct 9 12:26:40.846390183 (382h55m26.308482666s) [upload=336908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:26:40.846408758 (382h55m26.308501231s) [upload=336908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:26:41.094330866 (382h55m26.556423379s) [seg_pos={0 0}, upload=336908]: Begin upload segment... Oct 9 12:26:41.094403173 (382h55m26.556495646s) [seg_pos={0 0}, upload=336908]: Done begin upload segment. Oct 9 12:26:41.094419845 (382h55m26.556512329s) [noise=true , address=45.91.116.6:4974 , piece_id=V76LUWFCQP7T74HFEXHBMJXQZRSQSFOAX5A3GZT3MBQSIIUWONWA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.094439152 (382h55m26.556531635s) [noise=true , address=145.79.227.15:4876 , piece_id=XLUM526JLSMNMNDRDKBXG36B4G4KVL6WLQIJHVA5M4FWUMKXEEYQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.094889068 (382h55m26.556981552s) [noise=true , address=145.79.227.2:4115 , piece_id=X5QIBLDXTB6WNI5NM3TGZJO6CJQZLGGDCS6BEMTYARSCAOL6C5UQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.095193689 (382h55m26.557286172s) [noise=true , address=145.79.227.19:4078 , piece_id=F6OECBUFUU2PWSYMROTSJY47TCTHS2NJHID6IYOQHGMU3IGV5RXA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.095512736 (382h55m26.557605209s) [noise=true , address=145.79.227.12:4767 , piece_id=FK2KKBEU7VNJZVOWWQ2US5XVT5KJIVTOWG7POKPCJWQ5IQZZ6BFQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.095836142 (382h55m26.557928615s) [noise=true , address=145.79.227.18:4005 , piece_id=EPBY2RIJYADSKDSA7YRIDIRORFEH3TFNWSJU33PZL2P6U2HAI3HQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.096156573 (382h55m26.558249046s) [noise=true , address=145.79.227.11:4731 , piece_id=NZRQU3JCK46QBBOT6F5MSUYPF3LMWZ6LXSSVUSAOIRFACKJNEBPQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.096661644 (382h55m26.558754117s) [noise=true , address=45.91.116.9:4262 , piece_id=NPYFDG67EYKWI25Z5KC6RV4HA3SWK5HTHVFDKRLP3AL43AUPXA7Q, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.097164993 (382h55m26.559257476s) [noise=true , address=145.79.227.3:4150 , piece_id=3P7C3WB4WNGHZQYIVWNHJHOMQFBHAVHY7PNNUQWOFG7GFEVSTPGA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.097620761 (382h55m26.559713244s) [noise=true , address=45.91.116.7:4011 , piece_id=FOWPFEKBM3YUNYZELHYY2EIEMN65MJN3A6XYQRD3CDIKORRQAIKA, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.098718380 (382h55m26.560810863s) [noise=true , address=145.79.227.16:4909 , piece_id=ECE7RMOL4WFHTM2ETWD46FCGO4DSMGOKMLN4RD3LMF5E43AC3TIQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.099245084 (382h55m26.561337567s) [noise=true , address=77.90.31.17:4770 , piece_id=MSJ3XZFBOOPR33OXQN7FSBFTHXE5DPCYWRGMZA7YCUT2IIYVZ4OQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.099941009 (382h55m26.562033502s) [noise=true , address=145.79.227.17:4972 , piece_id=ITGSR44OGJUPEZLXNDRVNARQ57SZVBA3DSKG6VCAACLDYMDT6ZGQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.100621013 (382h55m26.562713497s) [noise=true , address=145.79.227.14:4840 , piece_id=QNUM5BN6X2CZF5G7OYWXC5X2OTH374A3OEWJTBU3NSBXG5PFRNKQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.102221601 (382h55m26.564314084s) [noise=true , address=145.79.227.2:4114 , piece_id=SMDLDOYCKUJNEXS3N7JUJECZEA74PJXXDPB2TAGWFSS5HJ4Y5V6A, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:41.104416040 (382h55m26.566508523s) [noise=true , address=77.90.31.16:4733 , piece_id=BLHAGWVLRELTP32R7SMCXOLGMWAJHGWXZ6XVUYQGDJ2BKZ55DEZQ, seg_pos={0 0}, upload=336908]: Uploading piece... Oct 9 12:26:42.944427443 (382h55m28.406519927s) [upload=336908]: Next returned nil segment Oct 9 12:26:42.944609239 (382h55m28.406701712s) [upload=336908]: Waiting for error group managing segments... Oct 9 12:26:43.308230354 (382h55m28.770322837s) [noise=true , address=45.91.116.6:4974 , piece_id=V76LUWFCQP7T74HFEXHBMJXQZRSQSFOAX5A3GZT3MBQSIIUWONWA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.310855472 (382h55m28.772947955s) [noise=true , address=77.90.31.16:4733 , piece_id=BLHAGWVLRELTP32R7SMCXOLGMWAJHGWXZ6XVUYQGDJ2BKZ55DEZQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.376033476 (382h55m28.838125959s) [noise=true , address=45.91.116.7:4011 , piece_id=FOWPFEKBM3YUNYZELHYY2EIEMN65MJN3A6XYQRD3CDIKORRQAIKA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.550644454 (382h55m29.012736937s) [noise=true , address=45.91.116.9:4262 , piece_id=NPYFDG67EYKWI25Z5KC6RV4HA3SWK5HTHVFDKRLP3AL43AUPXA7Q, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.600245505 (382h55m29.062337978s) [noise=true , address=77.90.31.17:4770 , piece_id=MSJ3XZFBOOPR33OXQN7FSBFTHXE5DPCYWRGMZA7YCUT2IIYVZ4OQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.731440649 (382h55m29.193533132s) [noise=true , address=145.79.227.12:4767 , piece_id=FK2KKBEU7VNJZVOWWQ2US5XVT5KJIVTOWG7POKPCJWQ5IQZZ6BFQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.798391097 (382h55m29.26048358s) [noise=true , address=145.79.227.19:4078 , piece_id=F6OECBUFUU2PWSYMROTSJY47TCTHS2NJHID6IYOQHGMU3IGV5RXA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.812053303 (382h55m29.274145786s) [noise=true , address=145.79.227.18:4005 , piece_id=EPBY2RIJYADSKDSA7YRIDIRORFEH3TFNWSJU33PZL2P6U2HAI3HQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.817079344 (382h55m29.279171837s) [noise=true , address=145.79.227.17:4972 , piece_id=ITGSR44OGJUPEZLXNDRVNARQ57SZVBA3DSKG6VCAACLDYMDT6ZGQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.836708482 (382h55m29.298800965s) [noise=true , address=145.79.227.2:4115 , piece_id=X5QIBLDXTB6WNI5NM3TGZJO6CJQZLGGDCS6BEMTYARSCAOL6C5UQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.918643605 (382h55m29.380736088s) [noise=true , address=145.79.227.3:4150 , piece_id=3P7C3WB4WNGHZQYIVWNHJHOMQFBHAVHY7PNNUQWOFG7GFEVSTPGA, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.922959764 (382h55m29.385052247s) [noise=true , address=145.79.227.11:4731 , piece_id=NZRQU3JCK46QBBOT6F5MSUYPF3LMWZ6LXSSVUSAOIRFACKJNEBPQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.927920290 (382h55m29.390012773s) [noise=true , address=145.79.227.14:4840 , piece_id=QNUM5BN6X2CZF5G7OYWXC5X2OTH374A3OEWJTBU3NSBXG5PFRNKQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: <nil> Oct 9 12:26:43.927944315 (382h55m29.390036788s) [seg_pos={0 0}, upload=336908]: Segment reached optimal threshold of 13 pieces. Oct 9 12:26:43.928058163 (382h55m29.390150646s) [noise=true , address=145.79.227.2:4114 , piece_id=SMDLDOYCKUJNEXS3N7JUJECZEA74PJXXDPB2TAGWFSS5HJ4Y5V6A, seg_pos={0 0}, upload=336908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: expected piece hash; context canceled; EOF Oct 9 12:26:43.928066528 (382h55m29.390159011s) [noise=true , address=145.79.227.2:4114 , piece_id=SMDLDOYCKUJNEXS3N7JUJECZEA74PJXXDPB2TAGWFSS5HJ4Y5V6A, seg_pos={0 0}, upload=336908]: Long tail cancellation due to optimal threshold Oct 9 12:26:43.928176477 (382h55m29.39026895s) [noise=true , address=145.79.227.15:4876 , piece_id=XLUM526JLSMNMNDRDKBXG36B4G4KVL6WLQIJHVA5M4FWUMKXEEYQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Oct 9 12:26:43.928184553 (382h55m29.390277026s) [noise=true , address=145.79.227.15:4876 , piece_id=XLUM526JLSMNMNDRDKBXG36B4G4KVL6WLQIJHVA5M4FWUMKXEEYQ, seg_pos={0 0}, upload=336908]: Long tail cancellation due to optimal threshold Oct 9 12:26:43.928292378 (382h55m29.390384871s) [noise=true , address=145.79.227.16:4909 , piece_id=ECE7RMOL4WFHTM2ETWD46FCGO4DSMGOKMLN4RD3LMF5E43AC3TIQ, seg_pos={0 0}, upload=336908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Oct 9 12:26:43.928300202 (382h55m29.390392686s) [noise=true , address=145.79.227.16:4909 , piece_id=ECE7RMOL4WFHTM2ETWD46FCGO4DSMGOKMLN4RD3LMF5E43AC3TIQ, seg_pos={0 0}, upload=336908]: Long tail cancellation due to optimal threshold Oct 9 12:26:43.928319760 (382h55m29.390412243s) [seg_pos={0 0}, upload=336908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:26:43.928402547 (382h55m29.39049503s) [upload=336908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:26:43.928410083 (382h55m29.390502566s) [upload=336908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:26:44.244007830 (382h55m29.706100333s) [upload=336908]: Done uploading segments... Oct 9 12:26:44.244032437 (382h55m29.70612492s) [upload=336908]: Upload finished. err: <nil> Duration: 30.007203551s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 8.810858754s FAIL (2025-10-09 12:26:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: Z6GDESV6ZH5WXCNL7RAHG245ISX4UHUHDZETFTR5NJ7L7RAWCC7A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: SZXMIT3UHRNAT6E2YKORYYSN6IBHVEOJKEWYIODVUCXITSM75RGQ) context deadline exceeded ======LOGS====== Oct 9 12:24:15.820257885 (382h53m1.282350358s) [upload=336873]: Starting upload Oct 9 12:24:15.820348929 (382h53m1.282441412s) [upload=336873]: Done starting upload Oct 9 12:24:15.820534422 (382h53m1.282626935s) [upload=336873]: Uploading segments... Oct 9 12:24:15.820846837 (382h53m1.28293935s) [upload=336873]: Got next segment. Inline: false Oct 9 12:24:15.820930407 (382h53m1.28302288s) [upload=336873]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:24:15.820939534 (382h53m1.283032017s) [upload=336873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:24:16.067767318 (382h53m1.529859791s) [seg_pos={0 0}, upload=336873]: Begin upload segment... Oct 9 12:24:16.067851938 (382h53m1.529944422s) [seg_pos={0 0}, upload=336873]: Done begin upload segment. Oct 9 12:24:16.067885362 (382h53m1.529977855s) [noise=true , address=77.90.31.11:4554 , piece_id=IWRVUZFF6RBTK5AOMJ3R2ATSFP7RSCF3YOEFMNYL33WWQTWVMLFA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.067895381 (382h53m1.529987864s) [noise=true , address=45.91.116.11:4118 , piece_id=5KCT72Z2SUYMFZL5PLR5UUQSSN55YD2Y2HQDTHI7FTW5Q7ZH642A, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.068970488 (382h53m1.531062971s) [noise=true , address=145.79.227.3:4150 , piece_id=DC25JXRBLL3RBIF4K5SLEWZJ3E6CN6WPJLEOY7SB6VCFJPYEZTQQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.070659014 (382h53m1.532751497s) [noise=true , address=45.91.116.12:4227 , piece_id=KAP3XLMG357W4OOBULPF6PQKRKUZKJW6PN534NLEEFI7VYE5J6DQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.071675760 (382h53m1.533768233s) [noise=true , address=145.79.227.16:4912 , piece_id=DHVG6G3BCR6JWYDBTCWVRRKBUVCFMBHFGPTTOMSKUXKHPJ2LT6OQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.074209665 (382h53m1.536302148s) [noise=true , address=77.90.31.12:4590 , piece_id=3JV6HD5FJX4DHTUTH4W3AUQYK7UKVCX67QBYRLDJ6KDC5IFIGFRQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.074581864 (382h53m1.536674367s) [noise=true , address=45.91.116.12:4226 , piece_id=Z6GDESV6ZH5WXCNL7RAHG245ISX4UHUHDZETFTR5NJ7L7RAWCC7A, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.076508332 (382h53m1.538600815s) [noise=true , address=45.91.116.10:4081 , piece_id=JL2DGIZEGC67GW5TTYQXMO6HVHA5RLTLZOLOFSE72H6ZNZ2RRBKQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.078003700 (382h53m1.540096173s) [noise=true , address=45.91.116.7:4009 , piece_id=R4EKYRWZVW3GD26SYQ67BWE64PMWNSB47KLSCAA6U5M7CPBKPFVQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.080947986 (382h53m1.54304048s) [noise=true , address=45.91.116.5:4298 , piece_id=SZXMIT3UHRNAT6E2YKORYYSN6IBHVEOJKEWYIODVUCXITSM75RGQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.084245375 (382h53m1.546337889s) [noise=true , address=145.79.227.11:4729 , piece_id=N3I73OTPJCVWUGTQADW4LEXCM3Y37RFLY7GLXTZI4OTNBEBCJJOA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.084890533 (382h53m1.546983017s) [noise=true , address=77.90.31.16:4733 , piece_id=2CW3U636EI4Q3LFQHQDAAYAOXESNR54Z6YTJEGOIVZEJA7QI7VOA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.087274864 (382h53m1.549367347s) [noise=true , address=145.79.227.17:4970 , piece_id=ZCIPU2STVY6FB3GE4RXLA7UCDXVW7FZSJWBU2IUXLOITOSLJMGSA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.088499526 (382h53m1.550592009s) [noise=true , address=45.91.116.13:4155 , piece_id=3PZNMLWBI6XPWEOM4P7YTJ2BDMLL3Q7SQNVJJ7L6RYVU3FAIKVVA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.091183778 (382h53m1.553276261s) [noise=true , address=77.90.31.19:4843 , piece_id=EWX4L32CTZUEWAH32GOWSU6WC3YZNXCPVDPQOXT53KAHB5745BQQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.093174298 (382h53m1.555266781s) [noise=true , address=77.90.31.10:4517 , piece_id=UHSBNXY7Q2JU3CIC44PAYC7QN26KXK3GUSDGM72E2HPSQMTJ24RQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:17.948665870 (382h53m3.410758343s) [upload=336873]: Next returned nil segment Oct 9 12:24:17.948686258 (382h53m3.410778741s) [upload=336873]: Waiting for error group managing segments... Oct 9 12:24:18.342625365 (382h53m3.804717858s) [noise=true , address=45.91.116.7:4009 , piece_id=R4EKYRWZVW3GD26SYQ67BWE64PMWNSB47KLSCAA6U5M7CPBKPFVQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.461585571 (382h53m3.923678054s) [noise=true , address=45.91.116.12:4226 , piece_id=Z6GDESV6ZH5WXCNL7RAHG245ISX4UHUHDZETFTR5NJ7L7RAWCC7A, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.480543130 (382h53m3.942635613s) [noise=true , address=45.91.116.12:4227 , piece_id=KAP3XLMG357W4OOBULPF6PQKRKUZKJW6PN534NLEEFI7VYE5J6DQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.483385242 (382h53m3.945477725s) [noise=true , address=77.90.31.11:4554 , piece_id=IWRVUZFF6RBTK5AOMJ3R2ATSFP7RSCF3YOEFMNYL33WWQTWVMLFA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.485441518 (382h53m3.947534001s) [noise=true , address=45.91.116.13:4155 , piece_id=3PZNMLWBI6XPWEOM4P7YTJ2BDMLL3Q7SQNVJJ7L6RYVU3FAIKVVA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.490915592 (382h53m3.953008075s) [noise=true , address=45.91.116.11:4118 , piece_id=5KCT72Z2SUYMFZL5PLR5UUQSSN55YD2Y2HQDTHI7FTW5Q7ZH642A, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.506184828 (382h53m3.968277311s) [noise=true , address=45.91.116.10:4081 , piece_id=JL2DGIZEGC67GW5TTYQXMO6HVHA5RLTLZOLOFSE72H6ZNZ2RRBKQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.514008676 (382h53m3.976101169s) [noise=true , address=77.90.31.10:4517 , piece_id=UHSBNXY7Q2JU3CIC44PAYC7QN26KXK3GUSDGM72E2HPSQMTJ24RQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.565897343 (382h53m4.027989826s) [noise=true , address=77.90.31.12:4590 , piece_id=3JV6HD5FJX4DHTUTH4W3AUQYK7UKVCX67QBYRLDJ6KDC5IFIGFRQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.566733996 (382h53m4.028826479s) [noise=true , address=77.90.31.16:4733 , piece_id=2CW3U636EI4Q3LFQHQDAAYAOXESNR54Z6YTJEGOIVZEJA7QI7VOA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.585137551 (382h53m4.047230024s) [noise=true , address=45.91.116.5:4298 , piece_id=SZXMIT3UHRNAT6E2YKORYYSN6IBHVEOJKEWYIODVUCXITSM75RGQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.863048544 (382h53m4.325141027s) [noise=true , address=145.79.227.3:4150 , piece_id=DC25JXRBLL3RBIF4K5SLEWZJ3E6CN6WPJLEOY7SB6VCFJPYEZTQQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.987687884 (382h53m4.449780368s) [noise=true , address=145.79.227.11:4729 , piece_id=N3I73OTPJCVWUGTQADW4LEXCM3Y37RFLY7GLXTZI4OTNBEBCJJOA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.987715417 (382h53m4.4498079s) [seg_pos={0 0}, upload=336873]: Segment reached optimal threshold of 13 pieces. Oct 9 12:24:18.987851416 (382h53m4.449943899s) [noise=true , address=145.79.227.16:4912 , piece_id=DHVG6G3BCR6JWYDBTCWVRRKBUVCFMBHFGPTTOMSKUXKHPJ2LT6OQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF Oct 9 12:24:18.987861014 (382h53m4.449953497s) [noise=true , address=145.79.227.16:4912 , piece_id=DHVG6G3BCR6JWYDBTCWVRRKBUVCFMBHFGPTTOMSKUXKHPJ2LT6OQ, seg_pos={0 0}, upload=336873]: Long tail cancellation due to optimal threshold Oct 9 12:24:18.987972416 (382h53m4.450064889s) [noise=true , address=145.79.227.17:4970 , piece_id=ZCIPU2STVY6FB3GE4RXLA7UCDXVW7FZSJWBU2IUXLOITOSLJMGSA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: expected piece hash; context canceled; EOF Oct 9 12:24:18.987978477 (382h53m4.450070961s) [noise=true , address=145.79.227.17:4970 , piece_id=ZCIPU2STVY6FB3GE4RXLA7UCDXVW7FZSJWBU2IUXLOITOSLJMGSA, seg_pos={0 0}, upload=336873]: Long tail cancellation due to optimal threshold Oct 9 12:24:18.988063871 (382h53m4.450156344s) [noise=true , address=77.90.31.19:4843 , piece_id=EWX4L32CTZUEWAH32GOWSU6WC3YZNXCPVDPQOXT53KAHB5745BQQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 9 12:24:18.988070323 (382h53m4.450162796s) [noise=true , address=77.90.31.19:4843 , piece_id=EWX4L32CTZUEWAH32GOWSU6WC3YZNXCPVDPQOXT53KAHB5745BQQ, seg_pos={0 0}, upload=336873]: Long tail cancellation due to optimal threshold Oct 9 12:24:18.988084289 (382h53m4.450176772s) [seg_pos={0 0}, upload=336873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:24:18.988155515 (382h53m4.450247998s) [upload=336873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:24:18.988160214 (382h53m4.450252697s) [upload=336873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:24:19.304120983 (382h53m4.766213456s) [upload=336873]: Done uploading segments... Oct 9 12:24:19.304144387 (382h53m4.76623687s) [upload=336873]: Upload finished. err: <nil> Duration: 8.810892398s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 8.208026093s FAIL (2025-10-09 12:25:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: Z6GDESV6ZH5WXCNL7RAHG245ISX4UHUHDZETFTR5NJ7L7RAWCC7A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: SZXMIT3UHRNAT6E2YKORYYSN6IBHVEOJKEWYIODVUCXITSM75RGQ) context deadline exceeded ======LOGS====== Oct 9 12:24:15.820257885 (382h53m1.282350358s) [upload=336873]: Starting upload Oct 9 12:24:15.820348929 (382h53m1.282441412s) [upload=336873]: Done starting upload Oct 9 12:24:15.820534422 (382h53m1.282626935s) [upload=336873]: Uploading segments... Oct 9 12:24:15.820846837 (382h53m1.28293935s) [upload=336873]: Got next segment. Inline: false Oct 9 12:24:15.820930407 (382h53m1.28302288s) [upload=336873]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:24:15.820939534 (382h53m1.283032017s) [upload=336873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:24:16.067767318 (382h53m1.529859791s) [seg_pos={0 0}, upload=336873]: Begin upload segment... Oct 9 12:24:16.067851938 (382h53m1.529944422s) [seg_pos={0 0}, upload=336873]: Done begin upload segment. Oct 9 12:24:16.067885362 (382h53m1.529977855s) [noise=true , address=77.90.31.11:4554 , piece_id=IWRVUZFF6RBTK5AOMJ3R2ATSFP7RSCF3YOEFMNYL33WWQTWVMLFA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.067895381 (382h53m1.529987864s) [noise=true , address=45.91.116.11:4118 , piece_id=5KCT72Z2SUYMFZL5PLR5UUQSSN55YD2Y2HQDTHI7FTW5Q7ZH642A, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.068970488 (382h53m1.531062971s) [noise=true , address=145.79.227.3:4150 , piece_id=DC25JXRBLL3RBIF4K5SLEWZJ3E6CN6WPJLEOY7SB6VCFJPYEZTQQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.070659014 (382h53m1.532751497s) [noise=true , address=45.91.116.12:4227 , piece_id=KAP3XLMG357W4OOBULPF6PQKRKUZKJW6PN534NLEEFI7VYE5J6DQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.071675760 (382h53m1.533768233s) [noise=true , address=145.79.227.16:4912 , piece_id=DHVG6G3BCR6JWYDBTCWVRRKBUVCFMBHFGPTTOMSKUXKHPJ2LT6OQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.074209665 (382h53m1.536302148s) [noise=true , address=77.90.31.12:4590 , piece_id=3JV6HD5FJX4DHTUTH4W3AUQYK7UKVCX67QBYRLDJ6KDC5IFIGFRQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.074581864 (382h53m1.536674367s) [noise=true , address=45.91.116.12:4226 , piece_id=Z6GDESV6ZH5WXCNL7RAHG245ISX4UHUHDZETFTR5NJ7L7RAWCC7A, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.076508332 (382h53m1.538600815s) [noise=true , address=45.91.116.10:4081 , piece_id=JL2DGIZEGC67GW5TTYQXMO6HVHA5RLTLZOLOFSE72H6ZNZ2RRBKQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.078003700 (382h53m1.540096173s) [noise=true , address=45.91.116.7:4009 , piece_id=R4EKYRWZVW3GD26SYQ67BWE64PMWNSB47KLSCAA6U5M7CPBKPFVQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.080947986 (382h53m1.54304048s) [noise=true , address=45.91.116.5:4298 , piece_id=SZXMIT3UHRNAT6E2YKORYYSN6IBHVEOJKEWYIODVUCXITSM75RGQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.084245375 (382h53m1.546337889s) [noise=true , address=145.79.227.11:4729 , piece_id=N3I73OTPJCVWUGTQADW4LEXCM3Y37RFLY7GLXTZI4OTNBEBCJJOA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.084890533 (382h53m1.546983017s) [noise=true , address=77.90.31.16:4733 , piece_id=2CW3U636EI4Q3LFQHQDAAYAOXESNR54Z6YTJEGOIVZEJA7QI7VOA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.087274864 (382h53m1.549367347s) [noise=true , address=145.79.227.17:4970 , piece_id=ZCIPU2STVY6FB3GE4RXLA7UCDXVW7FZSJWBU2IUXLOITOSLJMGSA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.088499526 (382h53m1.550592009s) [noise=true , address=45.91.116.13:4155 , piece_id=3PZNMLWBI6XPWEOM4P7YTJ2BDMLL3Q7SQNVJJ7L6RYVU3FAIKVVA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.091183778 (382h53m1.553276261s) [noise=true , address=77.90.31.19:4843 , piece_id=EWX4L32CTZUEWAH32GOWSU6WC3YZNXCPVDPQOXT53KAHB5745BQQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.093174298 (382h53m1.555266781s) [noise=true , address=77.90.31.10:4517 , piece_id=UHSBNXY7Q2JU3CIC44PAYC7QN26KXK3GUSDGM72E2HPSQMTJ24RQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:17.948665870 (382h53m3.410758343s) [upload=336873]: Next returned nil segment Oct 9 12:24:17.948686258 (382h53m3.410778741s) [upload=336873]: Waiting for error group managing segments... Oct 9 12:24:18.342625365 (382h53m3.804717858s) [noise=true , address=45.91.116.7:4009 , piece_id=R4EKYRWZVW3GD26SYQ67BWE64PMWNSB47KLSCAA6U5M7CPBKPFVQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.461585571 (382h53m3.923678054s) [noise=true , address=45.91.116.12:4226 , piece_id=Z6GDESV6ZH5WXCNL7RAHG245ISX4UHUHDZETFTR5NJ7L7RAWCC7A, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.480543130 (382h53m3.942635613s) [noise=true , address=45.91.116.12:4227 , piece_id=KAP3XLMG357W4OOBULPF6PQKRKUZKJW6PN534NLEEFI7VYE5J6DQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.483385242 (382h53m3.945477725s) [noise=true , address=77.90.31.11:4554 , piece_id=IWRVUZFF6RBTK5AOMJ3R2ATSFP7RSCF3YOEFMNYL33WWQTWVMLFA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.485441518 (382h53m3.947534001s) [noise=true , address=45.91.116.13:4155 , piece_id=3PZNMLWBI6XPWEOM4P7YTJ2BDMLL3Q7SQNVJJ7L6RYVU3FAIKVVA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.490915592 (382h53m3.953008075s) [noise=true , address=45.91.116.11:4118 , piece_id=5KCT72Z2SUYMFZL5PLR5UUQSSN55YD2Y2HQDTHI7FTW5Q7ZH642A, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.506184828 (382h53m3.968277311s) [noise=true , address=45.91.116.10:4081 , piece_id=JL2DGIZEGC67GW5TTYQXMO6HVHA5RLTLZOLOFSE72H6ZNZ2RRBKQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.514008676 (382h53m3.976101169s) [noise=true , address=77.90.31.10:4517 , piece_id=UHSBNXY7Q2JU3CIC44PAYC7QN26KXK3GUSDGM72E2HPSQMTJ24RQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.565897343 (382h53m4.027989826s) [noise=true , address=77.90.31.12:4590 , piece_id=3JV6HD5FJX4DHTUTH4W3AUQYK7UKVCX67QBYRLDJ6KDC5IFIGFRQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.566733996 (382h53m4.028826479s) [noise=true , address=77.90.31.16:4733 , piece_id=2CW3U636EI4Q3LFQHQDAAYAOXESNR54Z6YTJEGOIVZEJA7QI7VOA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.585137551 (382h53m4.047230024s) [noise=true , address=45.91.116.5:4298 , piece_id=SZXMIT3UHRNAT6E2YKORYYSN6IBHVEOJKEWYIODVUCXITSM75RGQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.863048544 (382h53m4.325141027s) [noise=true , address=145.79.227.3:4150 , piece_id=DC25JXRBLL3RBIF4K5SLEWZJ3E6CN6WPJLEOY7SB6VCFJPYEZTQQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.987687884 (382h53m4.449780368s) [noise=true , address=145.79.227.11:4729 , piece_id=N3I73OTPJCVWUGTQADW4LEXCM3Y37RFLY7GLXTZI4OTNBEBCJJOA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.987715417 (382h53m4.4498079s) [seg_pos={0 0}, upload=336873]: Segment reached optimal threshold of 13 pieces. Oct 9 12:24:18.987851416 (382h53m4.449943899s) [noise=true , address=145.79.227.16:4912 , piece_id=DHVG6G3BCR6JWYDBTCWVRRKBUVCFMBHFGPTTOMSKUXKHPJ2LT6OQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF Oct 9 12:24:18.987861014 (382h53m4.449953497s) [noise=true , address=145.79.227.16:4912 , piece_id=DHVG6G3BCR6JWYDBTCWVRRKBUVCFMBHFGPTTOMSKUXKHPJ2LT6OQ, seg_pos={0 0}, upload=336873]: Long tail cancellation due to optimal threshold Oct 9 12:24:18.987972416 (382h53m4.450064889s) [noise=true , address=145.79.227.17:4970 , piece_id=ZCIPU2STVY6FB3GE4RXLA7UCDXVW7FZSJWBU2IUXLOITOSLJMGSA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: expected piece hash; context canceled; EOF Oct 9 12:24:18.987978477 (382h53m4.450070961s) [noise=true , address=145.79.227.17:4970 , piece_id=ZCIPU2STVY6FB3GE4RXLA7UCDXVW7FZSJWBU2IUXLOITOSLJMGSA, seg_pos={0 0}, upload=336873]: Long tail cancellation due to optimal threshold Oct 9 12:24:18.988063871 (382h53m4.450156344s) [noise=true , address=77.90.31.19:4843 , piece_id=EWX4L32CTZUEWAH32GOWSU6WC3YZNXCPVDPQOXT53KAHB5745BQQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 9 12:24:18.988070323 (382h53m4.450162796s) [noise=true , address=77.90.31.19:4843 , piece_id=EWX4L32CTZUEWAH32GOWSU6WC3YZNXCPVDPQOXT53KAHB5745BQQ, seg_pos={0 0}, upload=336873]: Long tail cancellation due to optimal threshold Oct 9 12:24:18.988084289 (382h53m4.450176772s) [seg_pos={0 0}, upload=336873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:24:18.988155515 (382h53m4.450247998s) [upload=336873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:24:18.988160214 (382h53m4.450252697s) [upload=336873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:24:19.304120983 (382h53m4.766213456s) [upload=336873]: Done uploading segments... Oct 9 12:24:19.304144387 (382h53m4.76623687s) [upload=336873]: Upload finished. err: <nil> Duration: 8.208058694s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 14.335655292s FAIL (2025-10-09 12:25:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: Z6GDESV6ZH5WXCNL7RAHG245ISX4UHUHDZETFTR5NJ7L7RAWCC7A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: SZXMIT3UHRNAT6E2YKORYYSN6IBHVEOJKEWYIODVUCXITSM75RGQ) context deadline exceeded ======LOGS====== Oct 9 12:24:15.820257885 (382h53m1.282350358s) [upload=336873]: Starting upload Oct 9 12:24:15.820348929 (382h53m1.282441412s) [upload=336873]: Done starting upload Oct 9 12:24:15.820534422 (382h53m1.282626935s) [upload=336873]: Uploading segments... Oct 9 12:24:15.820846837 (382h53m1.28293935s) [upload=336873]: Got next segment. Inline: false Oct 9 12:24:15.820930407 (382h53m1.28302288s) [upload=336873]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:24:15.820939534 (382h53m1.283032017s) [upload=336873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:24:16.067767318 (382h53m1.529859791s) [seg_pos={0 0}, upload=336873]: Begin upload segment... Oct 9 12:24:16.067851938 (382h53m1.529944422s) [seg_pos={0 0}, upload=336873]: Done begin upload segment. Oct 9 12:24:16.067885362 (382h53m1.529977855s) [noise=true , address=77.90.31.11:4554 , piece_id=IWRVUZFF6RBTK5AOMJ3R2ATSFP7RSCF3YOEFMNYL33WWQTWVMLFA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.067895381 (382h53m1.529987864s) [noise=true , address=45.91.116.11:4118 , piece_id=5KCT72Z2SUYMFZL5PLR5UUQSSN55YD2Y2HQDTHI7FTW5Q7ZH642A, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.068970488 (382h53m1.531062971s) [noise=true , address=145.79.227.3:4150 , piece_id=DC25JXRBLL3RBIF4K5SLEWZJ3E6CN6WPJLEOY7SB6VCFJPYEZTQQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.070659014 (382h53m1.532751497s) [noise=true , address=45.91.116.12:4227 , piece_id=KAP3XLMG357W4OOBULPF6PQKRKUZKJW6PN534NLEEFI7VYE5J6DQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.071675760 (382h53m1.533768233s) [noise=true , address=145.79.227.16:4912 , piece_id=DHVG6G3BCR6JWYDBTCWVRRKBUVCFMBHFGPTTOMSKUXKHPJ2LT6OQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.074209665 (382h53m1.536302148s) [noise=true , address=77.90.31.12:4590 , piece_id=3JV6HD5FJX4DHTUTH4W3AUQYK7UKVCX67QBYRLDJ6KDC5IFIGFRQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.074581864 (382h53m1.536674367s) [noise=true , address=45.91.116.12:4226 , piece_id=Z6GDESV6ZH5WXCNL7RAHG245ISX4UHUHDZETFTR5NJ7L7RAWCC7A, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.076508332 (382h53m1.538600815s) [noise=true , address=45.91.116.10:4081 , piece_id=JL2DGIZEGC67GW5TTYQXMO6HVHA5RLTLZOLOFSE72H6ZNZ2RRBKQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.078003700 (382h53m1.540096173s) [noise=true , address=45.91.116.7:4009 , piece_id=R4EKYRWZVW3GD26SYQ67BWE64PMWNSB47KLSCAA6U5M7CPBKPFVQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.080947986 (382h53m1.54304048s) [noise=true , address=45.91.116.5:4298 , piece_id=SZXMIT3UHRNAT6E2YKORYYSN6IBHVEOJKEWYIODVUCXITSM75RGQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.084245375 (382h53m1.546337889s) [noise=true , address=145.79.227.11:4729 , piece_id=N3I73OTPJCVWUGTQADW4LEXCM3Y37RFLY7GLXTZI4OTNBEBCJJOA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.084890533 (382h53m1.546983017s) [noise=true , address=77.90.31.16:4733 , piece_id=2CW3U636EI4Q3LFQHQDAAYAOXESNR54Z6YTJEGOIVZEJA7QI7VOA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.087274864 (382h53m1.549367347s) [noise=true , address=145.79.227.17:4970 , piece_id=ZCIPU2STVY6FB3GE4RXLA7UCDXVW7FZSJWBU2IUXLOITOSLJMGSA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.088499526 (382h53m1.550592009s) [noise=true , address=45.91.116.13:4155 , piece_id=3PZNMLWBI6XPWEOM4P7YTJ2BDMLL3Q7SQNVJJ7L6RYVU3FAIKVVA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.091183778 (382h53m1.553276261s) [noise=true , address=77.90.31.19:4843 , piece_id=EWX4L32CTZUEWAH32GOWSU6WC3YZNXCPVDPQOXT53KAHB5745BQQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.093174298 (382h53m1.555266781s) [noise=true , address=77.90.31.10:4517 , piece_id=UHSBNXY7Q2JU3CIC44PAYC7QN26KXK3GUSDGM72E2HPSQMTJ24RQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:17.948665870 (382h53m3.410758343s) [upload=336873]: Next returned nil segment Oct 9 12:24:17.948686258 (382h53m3.410778741s) [upload=336873]: Waiting for error group managing segments... Oct 9 12:24:18.342625365 (382h53m3.804717858s) [noise=true , address=45.91.116.7:4009 , piece_id=R4EKYRWZVW3GD26SYQ67BWE64PMWNSB47KLSCAA6U5M7CPBKPFVQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.461585571 (382h53m3.923678054s) [noise=true , address=45.91.116.12:4226 , piece_id=Z6GDESV6ZH5WXCNL7RAHG245ISX4UHUHDZETFTR5NJ7L7RAWCC7A, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.480543130 (382h53m3.942635613s) [noise=true , address=45.91.116.12:4227 , piece_id=KAP3XLMG357W4OOBULPF6PQKRKUZKJW6PN534NLEEFI7VYE5J6DQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.483385242 (382h53m3.945477725s) [noise=true , address=77.90.31.11:4554 , piece_id=IWRVUZFF6RBTK5AOMJ3R2ATSFP7RSCF3YOEFMNYL33WWQTWVMLFA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.485441518 (382h53m3.947534001s) [noise=true , address=45.91.116.13:4155 , piece_id=3PZNMLWBI6XPWEOM4P7YTJ2BDMLL3Q7SQNVJJ7L6RYVU3FAIKVVA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.490915592 (382h53m3.953008075s) [noise=true , address=45.91.116.11:4118 , piece_id=5KCT72Z2SUYMFZL5PLR5UUQSSN55YD2Y2HQDTHI7FTW5Q7ZH642A, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.506184828 (382h53m3.968277311s) [noise=true , address=45.91.116.10:4081 , piece_id=JL2DGIZEGC67GW5TTYQXMO6HVHA5RLTLZOLOFSE72H6ZNZ2RRBKQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.514008676 (382h53m3.976101169s) [noise=true , address=77.90.31.10:4517 , piece_id=UHSBNXY7Q2JU3CIC44PAYC7QN26KXK3GUSDGM72E2HPSQMTJ24RQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.565897343 (382h53m4.027989826s) [noise=true , address=77.90.31.12:4590 , piece_id=3JV6HD5FJX4DHTUTH4W3AUQYK7UKVCX67QBYRLDJ6KDC5IFIGFRQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.566733996 (382h53m4.028826479s) [noise=true , address=77.90.31.16:4733 , piece_id=2CW3U636EI4Q3LFQHQDAAYAOXESNR54Z6YTJEGOIVZEJA7QI7VOA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.585137551 (382h53m4.047230024s) [noise=true , address=45.91.116.5:4298 , piece_id=SZXMIT3UHRNAT6E2YKORYYSN6IBHVEOJKEWYIODVUCXITSM75RGQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.863048544 (382h53m4.325141027s) [noise=true , address=145.79.227.3:4150 , piece_id=DC25JXRBLL3RBIF4K5SLEWZJ3E6CN6WPJLEOY7SB6VCFJPYEZTQQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.987687884 (382h53m4.449780368s) [noise=true , address=145.79.227.11:4729 , piece_id=N3I73OTPJCVWUGTQADW4LEXCM3Y37RFLY7GLXTZI4OTNBEBCJJOA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.987715417 (382h53m4.4498079s) [seg_pos={0 0}, upload=336873]: Segment reached optimal threshold of 13 pieces. Oct 9 12:24:18.987851416 (382h53m4.449943899s) [noise=true , address=145.79.227.16:4912 , piece_id=DHVG6G3BCR6JWYDBTCWVRRKBUVCFMBHFGPTTOMSKUXKHPJ2LT6OQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF Oct 9 12:24:18.987861014 (382h53m4.449953497s) [noise=true , address=145.79.227.16:4912 , piece_id=DHVG6G3BCR6JWYDBTCWVRRKBUVCFMBHFGPTTOMSKUXKHPJ2LT6OQ, seg_pos={0 0}, upload=336873]: Long tail cancellation due to optimal threshold Oct 9 12:24:18.987972416 (382h53m4.450064889s) [noise=true , address=145.79.227.17:4970 , piece_id=ZCIPU2STVY6FB3GE4RXLA7UCDXVW7FZSJWBU2IUXLOITOSLJMGSA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: expected piece hash; context canceled; EOF Oct 9 12:24:18.987978477 (382h53m4.450070961s) [noise=true , address=145.79.227.17:4970 , piece_id=ZCIPU2STVY6FB3GE4RXLA7UCDXVW7FZSJWBU2IUXLOITOSLJMGSA, seg_pos={0 0}, upload=336873]: Long tail cancellation due to optimal threshold Oct 9 12:24:18.988063871 (382h53m4.450156344s) [noise=true , address=77.90.31.19:4843 , piece_id=EWX4L32CTZUEWAH32GOWSU6WC3YZNXCPVDPQOXT53KAHB5745BQQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 9 12:24:18.988070323 (382h53m4.450162796s) [noise=true , address=77.90.31.19:4843 , piece_id=EWX4L32CTZUEWAH32GOWSU6WC3YZNXCPVDPQOXT53KAHB5745BQQ, seg_pos={0 0}, upload=336873]: Long tail cancellation due to optimal threshold Oct 9 12:24:18.988084289 (382h53m4.450176772s) [seg_pos={0 0}, upload=336873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:24:18.988155515 (382h53m4.450247998s) [upload=336873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:24:18.988160214 (382h53m4.450252697s) [upload=336873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:24:19.304120983 (382h53m4.766213456s) [upload=336873]: Done uploading segments... Oct 9 12:24:19.304144387 (382h53m4.76623687s) [upload=336873]: Upload finished. err: <nil> Duration: 14.335691422s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001858085s FAIL (2025-10-09 12:24:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: Z6GDESV6ZH5WXCNL7RAHG245ISX4UHUHDZETFTR5NJ7L7RAWCC7A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: SZXMIT3UHRNAT6E2YKORYYSN6IBHVEOJKEWYIODVUCXITSM75RGQ) context deadline exceeded ======LOGS====== Oct 9 12:24:15.820257885 (382h53m1.282350358s) [upload=336873]: Starting upload Oct 9 12:24:15.820348929 (382h53m1.282441412s) [upload=336873]: Done starting upload Oct 9 12:24:15.820534422 (382h53m1.282626935s) [upload=336873]: Uploading segments... Oct 9 12:24:15.820846837 (382h53m1.28293935s) [upload=336873]: Got next segment. Inline: false Oct 9 12:24:15.820930407 (382h53m1.28302288s) [upload=336873]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 12:24:15.820939534 (382h53m1.283032017s) [upload=336873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 12:24:16.067767318 (382h53m1.529859791s) [seg_pos={0 0}, upload=336873]: Begin upload segment... Oct 9 12:24:16.067851938 (382h53m1.529944422s) [seg_pos={0 0}, upload=336873]: Done begin upload segment. Oct 9 12:24:16.067885362 (382h53m1.529977855s) [noise=true , address=77.90.31.11:4554 , piece_id=IWRVUZFF6RBTK5AOMJ3R2ATSFP7RSCF3YOEFMNYL33WWQTWVMLFA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.067895381 (382h53m1.529987864s) [noise=true , address=45.91.116.11:4118 , piece_id=5KCT72Z2SUYMFZL5PLR5UUQSSN55YD2Y2HQDTHI7FTW5Q7ZH642A, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.068970488 (382h53m1.531062971s) [noise=true , address=145.79.227.3:4150 , piece_id=DC25JXRBLL3RBIF4K5SLEWZJ3E6CN6WPJLEOY7SB6VCFJPYEZTQQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.070659014 (382h53m1.532751497s) [noise=true , address=45.91.116.12:4227 , piece_id=KAP3XLMG357W4OOBULPF6PQKRKUZKJW6PN534NLEEFI7VYE5J6DQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.071675760 (382h53m1.533768233s) [noise=true , address=145.79.227.16:4912 , piece_id=DHVG6G3BCR6JWYDBTCWVRRKBUVCFMBHFGPTTOMSKUXKHPJ2LT6OQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.074209665 (382h53m1.536302148s) [noise=true , address=77.90.31.12:4590 , piece_id=3JV6HD5FJX4DHTUTH4W3AUQYK7UKVCX67QBYRLDJ6KDC5IFIGFRQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.074581864 (382h53m1.536674367s) [noise=true , address=45.91.116.12:4226 , piece_id=Z6GDESV6ZH5WXCNL7RAHG245ISX4UHUHDZETFTR5NJ7L7RAWCC7A, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.076508332 (382h53m1.538600815s) [noise=true , address=45.91.116.10:4081 , piece_id=JL2DGIZEGC67GW5TTYQXMO6HVHA5RLTLZOLOFSE72H6ZNZ2RRBKQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.078003700 (382h53m1.540096173s) [noise=true , address=45.91.116.7:4009 , piece_id=R4EKYRWZVW3GD26SYQ67BWE64PMWNSB47KLSCAA6U5M7CPBKPFVQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.080947986 (382h53m1.54304048s) [noise=true , address=45.91.116.5:4298 , piece_id=SZXMIT3UHRNAT6E2YKORYYSN6IBHVEOJKEWYIODVUCXITSM75RGQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.084245375 (382h53m1.546337889s) [noise=true , address=145.79.227.11:4729 , piece_id=N3I73OTPJCVWUGTQADW4LEXCM3Y37RFLY7GLXTZI4OTNBEBCJJOA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.084890533 (382h53m1.546983017s) [noise=true , address=77.90.31.16:4733 , piece_id=2CW3U636EI4Q3LFQHQDAAYAOXESNR54Z6YTJEGOIVZEJA7QI7VOA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.087274864 (382h53m1.549367347s) [noise=true , address=145.79.227.17:4970 , piece_id=ZCIPU2STVY6FB3GE4RXLA7UCDXVW7FZSJWBU2IUXLOITOSLJMGSA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.088499526 (382h53m1.550592009s) [noise=true , address=45.91.116.13:4155 , piece_id=3PZNMLWBI6XPWEOM4P7YTJ2BDMLL3Q7SQNVJJ7L6RYVU3FAIKVVA, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.091183778 (382h53m1.553276261s) [noise=true , address=77.90.31.19:4843 , piece_id=EWX4L32CTZUEWAH32GOWSU6WC3YZNXCPVDPQOXT53KAHB5745BQQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:16.093174298 (382h53m1.555266781s) [noise=true , address=77.90.31.10:4517 , piece_id=UHSBNXY7Q2JU3CIC44PAYC7QN26KXK3GUSDGM72E2HPSQMTJ24RQ, seg_pos={0 0}, upload=336873]: Uploading piece... Oct 9 12:24:17.948665870 (382h53m3.410758343s) [upload=336873]: Next returned nil segment Oct 9 12:24:17.948686258 (382h53m3.410778741s) [upload=336873]: Waiting for error group managing segments... Oct 9 12:24:18.342625365 (382h53m3.804717858s) [noise=true , address=45.91.116.7:4009 , piece_id=R4EKYRWZVW3GD26SYQ67BWE64PMWNSB47KLSCAA6U5M7CPBKPFVQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.461585571 (382h53m3.923678054s) [noise=true , address=45.91.116.12:4226 , piece_id=Z6GDESV6ZH5WXCNL7RAHG245ISX4UHUHDZETFTR5NJ7L7RAWCC7A, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.480543130 (382h53m3.942635613s) [noise=true , address=45.91.116.12:4227 , piece_id=KAP3XLMG357W4OOBULPF6PQKRKUZKJW6PN534NLEEFI7VYE5J6DQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.483385242 (382h53m3.945477725s) [noise=true , address=77.90.31.11:4554 , piece_id=IWRVUZFF6RBTK5AOMJ3R2ATSFP7RSCF3YOEFMNYL33WWQTWVMLFA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.485441518 (382h53m3.947534001s) [noise=true , address=45.91.116.13:4155 , piece_id=3PZNMLWBI6XPWEOM4P7YTJ2BDMLL3Q7SQNVJJ7L6RYVU3FAIKVVA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.490915592 (382h53m3.953008075s) [noise=true , address=45.91.116.11:4118 , piece_id=5KCT72Z2SUYMFZL5PLR5UUQSSN55YD2Y2HQDTHI7FTW5Q7ZH642A, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.506184828 (382h53m3.968277311s) [noise=true , address=45.91.116.10:4081 , piece_id=JL2DGIZEGC67GW5TTYQXMO6HVHA5RLTLZOLOFSE72H6ZNZ2RRBKQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.514008676 (382h53m3.976101169s) [noise=true , address=77.90.31.10:4517 , piece_id=UHSBNXY7Q2JU3CIC44PAYC7QN26KXK3GUSDGM72E2HPSQMTJ24RQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.565897343 (382h53m4.027989826s) [noise=true , address=77.90.31.12:4590 , piece_id=3JV6HD5FJX4DHTUTH4W3AUQYK7UKVCX67QBYRLDJ6KDC5IFIGFRQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.566733996 (382h53m4.028826479s) [noise=true , address=77.90.31.16:4733 , piece_id=2CW3U636EI4Q3LFQHQDAAYAOXESNR54Z6YTJEGOIVZEJA7QI7VOA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.585137551 (382h53m4.047230024s) [noise=true , address=45.91.116.5:4298 , piece_id=SZXMIT3UHRNAT6E2YKORYYSN6IBHVEOJKEWYIODVUCXITSM75RGQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.863048544 (382h53m4.325141027s) [noise=true , address=145.79.227.3:4150 , piece_id=DC25JXRBLL3RBIF4K5SLEWZJ3E6CN6WPJLEOY7SB6VCFJPYEZTQQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.987687884 (382h53m4.449780368s) [noise=true , address=145.79.227.11:4729 , piece_id=N3I73OTPJCVWUGTQADW4LEXCM3Y37RFLY7GLXTZI4OTNBEBCJJOA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: <nil> Oct 9 12:24:18.987715417 (382h53m4.4498079s) [seg_pos={0 0}, upload=336873]: Segment reached optimal threshold of 13 pieces. Oct 9 12:24:18.987851416 (382h53m4.449943899s) [noise=true , address=145.79.227.16:4912 , piece_id=DHVG6G3BCR6JWYDBTCWVRRKBUVCFMBHFGPTTOMSKUXKHPJ2LT6OQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF Oct 9 12:24:18.987861014 (382h53m4.449953497s) [noise=true , address=145.79.227.16:4912 , piece_id=DHVG6G3BCR6JWYDBTCWVRRKBUVCFMBHFGPTTOMSKUXKHPJ2LT6OQ, seg_pos={0 0}, upload=336873]: Long tail cancellation due to optimal threshold Oct 9 12:24:18.987972416 (382h53m4.450064889s) [noise=true , address=145.79.227.17:4970 , piece_id=ZCIPU2STVY6FB3GE4RXLA7UCDXVW7FZSJWBU2IUXLOITOSLJMGSA, seg_pos={0 0}, upload=336873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: expected piece hash; context canceled; EOF Oct 9 12:24:18.987978477 (382h53m4.450070961s) [noise=true , address=145.79.227.17:4970 , piece_id=ZCIPU2STVY6FB3GE4RXLA7UCDXVW7FZSJWBU2IUXLOITOSLJMGSA, seg_pos={0 0}, upload=336873]: Long tail cancellation due to optimal threshold Oct 9 12:24:18.988063871 (382h53m4.450156344s) [noise=true , address=77.90.31.19:4843 , piece_id=EWX4L32CTZUEWAH32GOWSU6WC3YZNXCPVDPQOXT53KAHB5745BQQ, seg_pos={0 0}, upload=336873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb): protocol: expected piece hash; context canceled; EOF Oct 9 12:24:18.988070323 (382h53m4.450162796s) [noise=true , address=77.90.31.19:4843 , piece_id=EWX4L32CTZUEWAH32GOWSU6WC3YZNXCPVDPQOXT53KAHB5745BQQ, seg_pos={0 0}, upload=336873]: Long tail cancellation due to optimal threshold Oct 9 12:24:18.988084289 (382h53m4.450176772s) [seg_pos={0 0}, upload=336873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 12:24:18.988155515 (382h53m4.450247998s) [upload=336873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 12:24:18.988160214 (382h53m4.450252697s) [upload=336873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 12:24:19.304120983 (382h53m4.766213456s) [upload=336873]: Done uploading segments... Oct 9 12:24:19.304144387 (382h53m4.76623687s) [upload=336873]: Upload finished. err: <nil> Duration: 30.00188732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 16.616943498s FAIL (2025-10-09 12:20:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 16.616974748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 12.970219s FAIL (2025-10-09 12:19:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 12.9702548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.754048093s FAIL (2025-10-09 12:18:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 6.754079474s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 9.152961625s FAIL (2025-10-09 12:17:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 9.152992153s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 22.574118017s FAIL (2025-10-09 12:17:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 22.574156711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 16.774471142s FAIL (2025-10-09 12:17:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 16.77453771s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 7.091996982s FAIL (2025-10-09 12:15:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 7.09203259s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 14.169878471s FAIL (2025-10-09 12:14:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 14.169902879s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 16.744687198s FAIL (2025-10-09 12:13:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 16.744717836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.881647796s FAIL (2025-10-09 12:12:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 6.881678737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 8.570467646s FAIL (2025-10-09 12:12:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 8.570501801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..........X................................................] duration: 8.126904849s FAIL (2025-10-09 12:11:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 8.126940125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX..........X...............................................] duration: 6.913254428s FAIL (2025-10-09 12:10:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 6.913278385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXX..........X..............................................] duration: 24.363740677s FAIL (2025-10-09 12:10:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 24.363788758s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XXX..........X.............................................] duration: 14.923009502s FAIL (2025-10-09 12:09:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 14.923040401s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XXX..........X............................................] duration: 6.99626893s FAIL (2025-10-09 12:08:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 6.996301073s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XXX..........X...........................................] duration: 12.306847105s FAIL (2025-10-09 12:08:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 12.306887721s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XXX..........X..........................................] duration: 8.579054792s FAIL (2025-10-09 12:07:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 8.579086581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XXX..........X.........................................] duration: 7.121173911s FAIL (2025-10-09 12:07:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 7.121197597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......XXX..........X........................................] duration: 29.65054328s FAIL (2025-10-09 12:07:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 9 11:36:22.445940477 (382h5m7.90803296s) [upload=336159]: Starting upload Oct 9 11:36:22.446011082 (382h5m7.908103565s) [upload=336159]: Done starting upload Oct 9 11:36:22.446152270 (382h5m7.908244743s) [upload=336159]: Uploading segments... Oct 9 11:36:22.446284502 (382h5m7.908376985s) [upload=336159]: Got next segment. Inline: false Oct 9 11:36:22.446321473 (382h5m7.908413956s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 9 11:36:22.446326723 (382h5m7.908419196s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 9 11:36:22.695206453 (382h5m8.157298946s) [seg_pos={0 0}, upload=336159]: Begin upload segment... Oct 9 11:36:22.695282688 (382h5m8.157375161s) [seg_pos={0 0}, upload=336159]: Done begin upload segment. Oct 9 11:36:22.695311944 (382h5m8.157404417s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.695396695 (382h5m8.157489198s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696050530 (382h5m8.158143003s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696215344 (382h5m8.158307817s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696654830 (382h5m8.158747313s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.696743490 (382h5m8.158835973s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697174610 (382h5m8.159267093s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.697631159 (382h5m8.159723642s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698042663 (382h5m8.160135146s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698323358 (382h5m8.160415841s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.698887693 (382h5m8.160980166s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699139372 (382h5m8.161231855s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699350604 (382h5m8.161443077s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699496422 (382h5m8.161588905s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.699894250 (382h5m8.161986733s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:22.700410974 (382h5m8.162503447s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Uploading piece... Oct 9 11:36:24.640724063 (382h5m10.102816546s) [upload=336159]: Next returned nil segment Oct 9 11:36:24.640747797 (382h5m10.10284027s) [upload=336159]: Waiting for error group managing segments... Oct 9 11:36:24.961649764 (382h5m10.423742247s) [noise=true , address=77.90.31.14:4662 , piece_id=QRWGCDFWNXVEVTD2ZCB3PQ5D3HWIU3P4PSGWBJXNKQEDUYSSTYWA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.087945327 (382h5m10.55003781s) [noise=true , address=77.90.31.16:4734 , piece_id=Y5B2IQDRHMOY736ON4GLXXSACBY5T736OYQVV7UTFEHMZRBF3T2Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.108980783 (382h5m10.571073266s) [noise=true , address=145.79.227.6:4257 , piece_id=LU7XLKHMKE6FCNK2O2GAME4DGK6MAZ75AEISCZELE3A7ONWSFJFQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.116550738 (382h5m10.578643211s) [noise=true , address=145.79.227.17:4970 , piece_id=EB6ENDUKAM64RZRQ4JIM6FNXCONWKAOZDISMT3MXQA3K5MEBFQCA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.163385721 (382h5m10.625478194s) [noise=true , address=45.91.116.7:4011 , piece_id=D6BQ2LOW7X5EWTHV6EKZ3FTBOMRI66ZJSHFS5S7TPWN7KNIPVBUA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.173742122 (382h5m10.635834605s) [noise=true , address=77.90.31.17:4770 , piece_id=5TXQW7PVPECWN37UHC756NA6MMR5B7CQHC7VOQQBCWXVTZZCSO7Q, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.201671667 (382h5m10.66376415s) [noise=true , address=45.91.116.12:4227 , piece_id=BPD7ICIQFHHCY5342WOGWHBRA325K35YHWKBUVQJ3BHZVMAE4ANA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.226597850 (382h5m10.688690333s) [noise=true , address=77.90.31.19:4841 , piece_id=6VJ4P4FTU5TGFGUU4BAQVKK3ZY4OALM3DJI3O7IGWAUPBBNTP6PQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.241504978 (382h5m10.703597471s) [noise=true , address=145.79.227.19:4079 , piece_id=CGIPZFWKQLBU2X72Y7FXNTOML6EAQVQTW6FVXIC5ICY742LLYMJQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242170585 (382h5m10.704263068s) [noise=true , address=145.79.227.7:4293 , piece_id=PD6Z3YDOWNTCQQNNGWZ2TKNOD4KUVLJZPZF7IKVFIQAP7LKR3APQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242181045 (382h5m10.704273528s) [noise=true , address=77.90.31.11:4555 , piece_id=VMOARACGP57XHXETACPHAAMN2NV3F26HLPJRWOSWJPNHGK43O7EQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.242784353 (382h5m10.704876836s) [noise=true , address=145.79.227.18:4005 , piece_id=KCM4IVDYWOJ7DUK5ESBKGN4Q7BSHJG25L6CWB3IFKGGKYISB2GHA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243338939 (382h5m10.705431422s) [noise=true , address=145.79.227.5:4223 , piece_id=SNVELEV464X7ELMJEWW6HIXIAX5IRXWDDLNTTH7QCAUC5UDQWZPQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: <nil> Oct 9 11:36:25.243366702 (382h5m10.705459185s) [seg_pos={0 0}, upload=336159]: Segment reached optimal threshold of 13 pieces. Oct 9 11:36:25.243509975 (382h5m10.705602448s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243520345 (382h5m10.705612828s) [noise=true , address=145.79.227.3:4150 , piece_id=XAPNHIS2AR7WRGOJ37HKRJK54O6WWSVBJIFJSK3PK4XIOWZZPNSQ, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243562085 (382h5m10.705654558s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243568587 (382h5m10.70566107s) [noise=true , address=77.90.31.10:4519 , piece_id=D2Z7XD6JXK4NQ4JKJ4ZILWIBX5E5ORFAWWCRQEMWG7I2MWVH2TKA, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243582693 (382h5m10.705675187s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 9 11:36:25.243594757 (382h5m10.70568724s) [noise=true , address=45.91.116.5:4298 , piece_id=3LQGC55GX7VDCWF6U3VN2GJJ7SVJHYYPBFH7KZYAPHA7CY5L3A7A, seg_pos={0 0}, upload=336159]: Long tail cancellation due to optimal threshold Oct 9 11:36:25.243614805 (382h5m10.705707288s) [seg_pos={0 0}, upload=336159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 9 11:36:25.243692833 (382h5m10.705785316s) [upload=336159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 9 11:36:25.243699196 (382h5m10.705791679s) [upload=336159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 9 11:36:25.556395616 (382h5m11.018488109s) [upload=336159]: Done uploading segments... Oct 9 11:36:25.556421055 (382h5m11.018513538s) [upload=336159]: Upload finished. err: <nil> Duration: 29.650578196s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)