Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................................X.XXX...X...........] duration: 6.385753207s FAIL (2025-10-07 01:01:40 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.385776092s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................................X..XX..XX...X..X] duration: 30.002397277s FAIL (2025-10-07 01:01:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: TQ5EOVF2WRBCJACPJM3737IOCFAR6NAYWDYIVUFZY6U2YU3JO2SA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: HMMJUK3GQH73MREUCXLUQJBHTZAOADUJCVWFCR3AJBTCP4GRO5AQ) context deadline exceeded ======LOGS====== ct 7 01:01:10.121839009 (323h29m55.583931492s) [upload=284702]: Starting upload Oct 7 01:01:10.121929321 (323h29m55.584021804s) [upload=284702]: Done starting upload Oct 7 01:01:10.122196790 (323h29m55.584289273s) [upload=284702]: Uploading segments... Oct 7 01:01:10.122261994 (323h29m55.584354478s) [upload=284702]: Got next segment. Inline: false Oct 7 01:01:10.122302772 (323h29m55.584395255s) [upload=284702]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 01:01:10.122308583 (323h29m55.584401076s) [upload=284702]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 01:01:10.411338415 (323h29m55.873430898s) [seg_pos={0 0}, upload=284702]: Begin upload segment... Oct 7 01:01:10.411397707 (323h29m55.87349019s) [seg_pos={0 0}, upload=284702]: Done begin upload segment. Oct 7 01:01:10.411423877 (323h29m55.87351635s) [noise=true , address=145.79.227.4:4186 , piece_id=N7TEHBR7IWNLBUILH332AC7WNSJNZ4XYDS7AX5GCECPCPI7FB4HA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.411471208 (323h29m55.873563681s) [noise=true , address=145.79.227.14:4839 , piece_id=LNOEB3X2XDTT47M6GATWJWVYA3R62L5XMXLZKEJ374GKPUM4KTRQ, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.412301959 (323h29m55.874394442s) [noise=true , address=145.79.227.19:4077 , piece_id=CLZNAOMAJXXZLVODTRFFOL52LUO5BQQJL224AMBPHGWT4AMSME3A, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413077777 (323h29m55.87517025s) [noise=true , address=45.91.116.5:4299 , piece_id=TQ5EOVF2WRBCJACPJM3737IOCFAR6NAYWDYIVUFZY6U2YU3JO2SA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413115959 (323h29m55.875208432s) [noise=true , address=77.90.31.14:4663 , piece_id=HMMJUK3GQH73MREUCXLUQJBHTZAOADUJCVWFCR3AJBTCP4GRO5AQ, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413422523 (323h29m55.875515006s) [noise=true , address=77.90.31.16:4733 , piece_id=CKP2NN7B7J2PAGORMCQ42MGISFEQQUQLIQR32CVD5YBVLVC2L4YA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413833645 (323h29m55.875926128s) [noise=true , address=77.90.31.18:4806 , piece_id=E7VGWTIYKDDKBRKWUGAINRUEJ47EBJYKEGS5X5HNOCKHZT5ZR45A, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.413966208 (323h29m55.876058691s) [noise=true , address=145.79.227.17:4972 , piece_id=GTLUNX4O7C5EEIUP25RZJWGQ2OBTHOBBZDWM76QAQMCRLTRR2N2Q, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.414258555 (323h29m55.876351028s) [noise=true , address=145.79.227.7:4293 , piece_id=OQQFJVQBPFQPMKCJOJLPK3VWT5ZQRZCFNOGPWOGX4ZD7BXKM6GHA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.414710566 (323h29m55.876803049s) [noise=true , address=77.90.31.11:4553 , piece_id=3NDPW2SUKFOGTUHP7C2OOICBT6DND77SHIN6GX3SBUOJCQKKHNMA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.414886451 (323h29m55.876978924s) [noise=true , address=145.79.227.15:4873 , piece_id=RALEQ5GZAQ4UFZUKOCQZLLWOJBXWXP5A2RDQHNCO226NK5WFAYQA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.415062666 (323h29m55.877155149s) [noise=true , address=145.79.227.3:4150 , piece_id=KCZXJ26XHDVDXR7R5NVZKJEASLQDH4ASUZE2CYHSJWW7MN3J5LHQ, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.415419926 (323h29m55.877512409s) [noise=true , address=145.79.227.13:4803 , piece_id=7KCTQU6ZBG6FZQHSQN64PXJUXDNCHUCDQRFUNCC5NNJL3MF4FQSQ, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.415697455 (323h29m55.877789948s) [noise=true , address=145.79.227.19:4078 , piece_id=5CBQOJN4YUTUONPOXK2TNPNYLBEYPBJJW3UFZ4Z55K2TFJ6L2NCA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.416700565 (323h29m55.878793038s) [noise=true , address=77.90.31.19:4842 , piece_id=NNKDBWZPJJEB7SLFEBUP5LO4HBHQ7MSPPUHV6CQIDRIHJV6OGS6A, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:10.417369609 (323h29m55.879462092s) [noise=true , address=45.91.116.12:4226 , piece_id=V63GMYNF6RXLUHYPLSYX4CGQF4BBPF7IOFVXSFT4N77T7RCGY4LA, seg_pos={0 0}, upload=284702]: Uploading piece... Oct 7 01:01:12.119016842 (323h29m57.581109325s) [upload=284702]: Next returned nil segment Oct 7 01:01:12.119114698 (323h29m57.581207201s) [upload=284702]: Waiting for error group managing segments... Oct 7 01:01:12.398844259 (323h29m57.860936742s) [noise=true , address=145.79.227.19:4077 , piece_id=CLZNAOMAJXXZLVODTRFFOL52LUO5BQQJL224AMBPHGWT4AMSME3A, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.402151106 (323h29m57.864243579s) [noise=true , address=145.79.227.14:4839 , piece_id=LNOEB3X2XDTT47M6GATWJWVYA3R62L5XMXLZKEJ374GKPUM4KTRQ, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.405187528 (323h29m57.867280011s) [noise=true , address=145.79.227.3:4150 , piece_id=KCZXJ26XHDVDXR7R5NVZKJEASLQDH4ASUZE2CYHSJWW7MN3J5LHQ, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.421155123 (323h29m57.883247596s) [noise=true , address=145.79.227.15:4873 , piece_id=RALEQ5GZAQ4UFZUKOCQZLLWOJBXWXP5A2RDQHNCO226NK5WFAYQA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.421662399 (323h29m57.883754883s) [noise=true , address=145.79.227.13:4803 , piece_id=7KCTQU6ZBG6FZQHSQN64PXJUXDNCHUCDQRFUNCC5NNJL3MF4FQSQ, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.423239743 (323h29m57.885332226s) [noise=true , address=145.79.227.17:4972 , piece_id=GTLUNX4O7C5EEIUP25RZJWGQ2OBTHOBBZDWM76QAQMCRLTRR2N2Q, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.425523362 (323h29m57.887615835s) [noise=true , address=145.79.227.19:4078 , piece_id=5CBQOJN4YUTUONPOXK2TNPNYLBEYPBJJW3UFZ4Z55K2TFJ6L2NCA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.425952498 (323h29m57.888044971s) [noise=true , address=145.79.227.4:4186 , piece_id=N7TEHBR7IWNLBUILH332AC7WNSJNZ4XYDS7AX5GCECPCPI7FB4HA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.431251198 (323h29m57.893343681s) [noise=true , address=145.79.227.7:4293 , piece_id=OQQFJVQBPFQPMKCJOJLPK3VWT5ZQRZCFNOGPWOGX4ZD7BXKM6GHA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.636922390 (323h29m58.099014873s) [noise=true , address=77.90.31.11:4553 , piece_id=3NDPW2SUKFOGTUHP7C2OOICBT6DND77SHIN6GX3SBUOJCQKKHNMA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.683157060 (323h29m58.145249543s) [noise=true , address=77.90.31.19:4842 , piece_id=NNKDBWZPJJEB7SLFEBUP5LO4HBHQ7MSPPUHV6CQIDRIHJV6OGS6A, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.683342783 (323h29m58.145435256s) [noise=true , address=45.91.116.5:4299 , piece_id=TQ5EOVF2WRBCJACPJM3737IOCFAR6NAYWDYIVUFZY6U2YU3JO2SA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.709183959 (323h29m58.171276442s) [noise=true , address=77.90.31.14:4663 , piece_id=HMMJUK3GQH73MREUCXLUQJBHTZAOADUJCVWFCR3AJBTCP4GRO5AQ, seg_pos={0 0}, upload=284702]: Done uploading piece. err: <nil> Oct 7 01:01:12.709211742 (323h29m58.171304235s) [seg_pos={0 0}, upload=284702]: Segment reached optimal threshold of 13 pieces. Oct 7 01:01:12.709372308 (323h29m58.171464791s) [noise=true , address=77.90.31.18:4806 , piece_id=E7VGWTIYKDDKBRKWUGAINRUEJ47EBJYKEGS5X5HNOCKHZT5ZR45A, seg_pos={0 0}, upload=284702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:38918->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:38918->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38918->77.90.31.18:4806: use of closed network connection Oct 7 01:01:12.709384090 (323h29m58.171476573s) [noise=true , address=77.90.31.18:4806 , piece_id=E7VGWTIYKDDKBRKWUGAINRUEJ47EBJYKEGS5X5HNOCKHZT5ZR45A, seg_pos={0 0}, upload=284702]: Long tail cancellation due to optimal threshold Oct 7 01:01:12.709527122 (323h29m58.171619595s) [noise=true , address=45.91.116.12:4226 , piece_id=V63GMYNF6RXLUHYPLSYX4CGQF4BBPF7IOFVXSFT4N77T7RCGY4LA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Oct 7 01:01:12.709535739 (323h29m58.171628222s) [noise=true , address=45.91.116.12:4226 , piece_id=V63GMYNF6RXLUHYPLSYX4CGQF4BBPF7IOFVXSFT4N77T7RCGY4LA, seg_pos={0 0}, upload=284702]: Long tail cancellation due to optimal threshold Oct 7 01:01:12.709614038 (323h29m58.171706521s) [noise=true , address=77.90.31.16:4733 , piece_id=CKP2NN7B7J2PAGORMCQ42MGISFEQQUQLIQR32CVD5YBVLVC2L4YA, seg_pos={0 0}, upload=284702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 7 01:01:12.709624408 (323h29m58.171716891s) [noise=true , address=77.90.31.16:4733 , piece_id=CKP2NN7B7J2PAGORMCQ42MGISFEQQUQLIQR32CVD5YBVLVC2L4YA, seg_pos={0 0}, upload=284702]: Long tail cancellation due to optimal threshold Oct 7 01:01:12.709639517 (323h29m58.171732s) [seg_pos={0 0}, upload=284702]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 01:01:12.709732493 (323h29m58.171824976s) [upload=284702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 01:01:12.709738976 (323h29m58.171831459s) [upload=284702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 01:01:13.360148075 (323h29m58.822240558s) [upload=284702]: Done uploading segments... Oct 7 01:01:13.360170197 (323h29m58.82226267s) [upload=284702]: Upload finished. err: <nil> Duration: 30.002422266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 31.960621015s FAIL (2025-10-07 01:01:37 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 31.960654528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................................X.XXX...X..........] duration: 21.496968072s FAIL (2025-10-07 01:01:33 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 21.49699802s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................................X.XXX...X.........] duration: 17.563188905s FAIL (2025-10-07 01:01:09 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 17.563214292s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................................X..XX..XX...X..] duration: 9.164958742s FAIL (2025-10-07 01:00:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q) context deadline exceeded; piecestore: (Node ID: 12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, Piece ID: HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ) context deadline exceeded ======LOGS====== Oct 7 00:59:12.685186317 (323h27m58.14727879s) [upload=284675]: Starting upload Oct 7 00:59:12.685278293 (323h27m58.147370776s) [upload=284675]: Done starting upload Oct 7 00:59:12.685454518 (323h27m58.147547001s) [upload=284675]: Uploading segments... Oct 7 00:59:12.685581469 (323h27m58.147673942s) [upload=284675]: Got next segment. Inline: false Oct 7 00:59:12.685608781 (323h27m58.147701265s) [upload=284675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:59:12.685613420 (323h27m58.147705903s) [upload=284675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:59:13.016521515 (323h27m58.478613998s) [seg_pos={0 0}, upload=284675]: Begin upload segment... Oct 7 00:59:13.016598392 (323h27m58.478690875s) [seg_pos={0 0}, upload=284675]: Done begin upload segment. Oct 7 00:59:13.016636374 (323h27m58.478728857s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016653387 (323h27m58.47874586s) [noise=true , address=145.79.227.19:4077 , piece_id=URF6Q57LGL3IB5CGYBSRFXA7I7MQH7TGFR2LWXPL7AJJP34SCR4Q, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016711817 (323h27m58.4788043s) [noise=true , address=145.79.227.13:4802 , piece_id=4JLFWK3TBNSL24R7TMCTTCHTC4IJYJXZ7MNEEU53TEJNMZGCR6IQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016777923 (323h27m58.478870406s) [noise=true , address=45.91.116.14:4189 , piece_id=22YZD55HNXA42ICR3R2YBA2GW7HPSEERM3DZZTXG3KOQ52IPCJWQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017181683 (323h27m58.479274166s) [noise=true , address=145.79.227.14:4838 , piece_id=6WPMFZ3POXZ2O7OJSEOYI7JH77AVHA36IKZV4FBVKARJXOIUP6TA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017556496 (323h27m58.47964897s) [noise=true , address=145.79.227.5:4222 , piece_id=KOAPDQKMHV3AGUVR5K7C3XETOXVRQETK4YBKL3F3V52PPNK6ZZAA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017921681 (323h27m58.480014164s) [noise=true , address=145.79.227.6:4258 , piece_id=KTKXNLQEOA4YMN2J67SVIEHJ7SJAIMWD2WGEJ2J2RW3ODQQBJFAQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.018279232 (323h27m58.480371715s) [noise=true , address=145.79.227.6:4257 , piece_id=2VDMEANJFQ3XQUU5KYA3K6M3KJ57JS4ECMDVWWMEBBCWPQ46HXIA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.018971991 (323h27m58.481064474s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.019174026 (323h27m58.481266499s) [noise=true , address=45.91.116.13:4153 , piece_id=G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.019858951 (323h27m58.481951424s) [noise=true , address=145.79.227.17:4969 , piece_id=34HLNC3V7GXQU4PY6GUHWJP5UNNHPRMBB2NLUAYV7EUUAWKWCPUQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.020800053 (323h27m58.482892536s) [noise=true , address=77.90.31.17:4771 , piece_id=HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.020990314 (323h27m58.483082807s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.022234434 (323h27m58.484326917s) [noise=true , address=77.90.31.17:4770 , piece_id=OA5BNZNICFW4ALFRXZ5UJX5KA6YQJJ5MFHJHRFX4Q23FUI3LQTQA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.025144545 (323h27m58.487237029s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y6W4Z6X6RO25JNYNRCY66VJCN5JSCDKHOJVQTDRKMKO2RWCGHVQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.027054692 (323h27m58.489147175s) [noise=true , address=145.79.227.4:4185 , piece_id=NGXAZCDFBWCQVL7RMLRHRVA3QQL23JCG5YR6VM5K5GPGQWA5YL2A, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:14.718509397 (323h28m0.18060188s) [upload=284675]: Next returned nil segment Oct 7 00:59:14.718532982 (323h28m0.180625465s) [upload=284675]: Waiting for error group managing segments... Oct 7 00:59:14.997286224 (323h28m0.459378697s) [noise=true , address=145.79.227.5:4222 , piece_id=KOAPDQKMHV3AGUVR5K7C3XETOXVRQETK4YBKL3F3V52PPNK6ZZAA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.005665701 (323h28m0.467758194s) [noise=true , address=145.79.227.6:4257 , piece_id=2VDMEANJFQ3XQUU5KYA3K6M3KJ57JS4ECMDVWWMEBBCWPQ46HXIA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.012184233 (323h28m0.474276716s) [noise=true , address=145.79.227.14:4838 , piece_id=6WPMFZ3POXZ2O7OJSEOYI7JH77AVHA36IKZV4FBVKARJXOIUP6TA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.015080098 (323h28m0.477172581s) [noise=true , address=145.79.227.6:4258 , piece_id=KTKXNLQEOA4YMN2J67SVIEHJ7SJAIMWD2WGEJ2J2RW3ODQQBJFAQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.015967287 (323h28m0.47805976s) [noise=true , address=145.79.227.17:4969 , piece_id=34HLNC3V7GXQU4PY6GUHWJP5UNNHPRMBB2NLUAYV7EUUAWKWCPUQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.017137155 (323h28m0.479229628s) [noise=true , address=145.79.227.13:4802 , piece_id=4JLFWK3TBNSL24R7TMCTTCHTC4IJYJXZ7MNEEU53TEJNMZGCR6IQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.033318026 (323h28m0.495410509s) [noise=true , address=145.79.227.4:4185 , piece_id=NGXAZCDFBWCQVL7RMLRHRVA3QQL23JCG5YR6VM5K5GPGQWA5YL2A, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.057300663 (323h28m0.519393136s) [noise=true , address=145.79.227.19:4077 , piece_id=URF6Q57LGL3IB5CGYBSRFXA7I7MQH7TGFR2LWXPL7AJJP34SCR4Q, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.174223514 (323h28m0.636315997s) [noise=true , address=45.91.116.14:4189 , piece_id=22YZD55HNXA42ICR3R2YBA2GW7HPSEERM3DZZTXG3KOQ52IPCJWQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.247155398 (323h28m0.709247871s) [noise=true , address=45.91.116.13:4153 , piece_id=G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.247337565 (323h28m0.709430048s) [noise=true , address=77.90.31.17:4770 , piece_id=OA5BNZNICFW4ALFRXZ5UJX5KA6YQJJ5MFHJHRFX4Q23FUI3LQTQA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.314344858 (323h28m0.776437341s) [noise=true , address=77.90.31.17:4771 , piece_id=HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.323081303 (323h28m0.785173787s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y6W4Z6X6RO25JNYNRCY66VJCN5JSCDKHOJVQTDRKMKO2RWCGHVQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.323112513 (323h28m0.785204996s) [seg_pos={0 0}, upload=284675]: Segment reached optimal threshold of 13 pieces. Oct 7 00:59:15.323280172 (323h28m0.785372655s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323294269 (323h28m0.785386742s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323360255 (323h28m0.785452728s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323367799 (323h28m0.785460282s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323548955 (323h28m0.785641438s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323620661 (323h28m0.785713144s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323725931 (323h28m0.785818414s) [seg_pos={0 0}, upload=284675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:59:15.323845278 (323h28m0.785937751s) [upload=284675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:59:15.323856119 (323h28m0.785948602s) [upload=284675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:59:15.677146474 (323h28m1.139238947s) [upload=284675]: Done uploading segments... Oct 7 00:59:15.677173235 (323h28m1.139265718s) [upload=284675]: Upload finished. err: <nil> Duration: 9.16498928s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................................X.XXX...X........] duration: 9.488944926s FAIL (2025-10-07 01:00:31 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 9.488973642s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.085871393s FAIL (2025-10-07 01:00:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 6.085904757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................................X..XX..XX...X.] duration: 6.210321006s FAIL (2025-10-07 01:00:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q) context deadline exceeded; piecestore: (Node ID: 12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, Piece ID: HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ) context deadline exceeded ======LOGS====== Oct 7 00:59:12.685186317 (323h27m58.14727879s) [upload=284675]: Starting upload Oct 7 00:59:12.685278293 (323h27m58.147370776s) [upload=284675]: Done starting upload Oct 7 00:59:12.685454518 (323h27m58.147547001s) [upload=284675]: Uploading segments... Oct 7 00:59:12.685581469 (323h27m58.147673942s) [upload=284675]: Got next segment. Inline: false Oct 7 00:59:12.685608781 (323h27m58.147701265s) [upload=284675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:59:12.685613420 (323h27m58.147705903s) [upload=284675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:59:13.016521515 (323h27m58.478613998s) [seg_pos={0 0}, upload=284675]: Begin upload segment... Oct 7 00:59:13.016598392 (323h27m58.478690875s) [seg_pos={0 0}, upload=284675]: Done begin upload segment. Oct 7 00:59:13.016636374 (323h27m58.478728857s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016653387 (323h27m58.47874586s) [noise=true , address=145.79.227.19:4077 , piece_id=URF6Q57LGL3IB5CGYBSRFXA7I7MQH7TGFR2LWXPL7AJJP34SCR4Q, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016711817 (323h27m58.4788043s) [noise=true , address=145.79.227.13:4802 , piece_id=4JLFWK3TBNSL24R7TMCTTCHTC4IJYJXZ7MNEEU53TEJNMZGCR6IQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016777923 (323h27m58.478870406s) [noise=true , address=45.91.116.14:4189 , piece_id=22YZD55HNXA42ICR3R2YBA2GW7HPSEERM3DZZTXG3KOQ52IPCJWQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017181683 (323h27m58.479274166s) [noise=true , address=145.79.227.14:4838 , piece_id=6WPMFZ3POXZ2O7OJSEOYI7JH77AVHA36IKZV4FBVKARJXOIUP6TA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017556496 (323h27m58.47964897s) [noise=true , address=145.79.227.5:4222 , piece_id=KOAPDQKMHV3AGUVR5K7C3XETOXVRQETK4YBKL3F3V52PPNK6ZZAA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017921681 (323h27m58.480014164s) [noise=true , address=145.79.227.6:4258 , piece_id=KTKXNLQEOA4YMN2J67SVIEHJ7SJAIMWD2WGEJ2J2RW3ODQQBJFAQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.018279232 (323h27m58.480371715s) [noise=true , address=145.79.227.6:4257 , piece_id=2VDMEANJFQ3XQUU5KYA3K6M3KJ57JS4ECMDVWWMEBBCWPQ46HXIA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.018971991 (323h27m58.481064474s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.019174026 (323h27m58.481266499s) [noise=true , address=45.91.116.13:4153 , piece_id=G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.019858951 (323h27m58.481951424s) [noise=true , address=145.79.227.17:4969 , piece_id=34HLNC3V7GXQU4PY6GUHWJP5UNNHPRMBB2NLUAYV7EUUAWKWCPUQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.020800053 (323h27m58.482892536s) [noise=true , address=77.90.31.17:4771 , piece_id=HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.020990314 (323h27m58.483082807s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.022234434 (323h27m58.484326917s) [noise=true , address=77.90.31.17:4770 , piece_id=OA5BNZNICFW4ALFRXZ5UJX5KA6YQJJ5MFHJHRFX4Q23FUI3LQTQA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.025144545 (323h27m58.487237029s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y6W4Z6X6RO25JNYNRCY66VJCN5JSCDKHOJVQTDRKMKO2RWCGHVQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.027054692 (323h27m58.489147175s) [noise=true , address=145.79.227.4:4185 , piece_id=NGXAZCDFBWCQVL7RMLRHRVA3QQL23JCG5YR6VM5K5GPGQWA5YL2A, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:14.718509397 (323h28m0.18060188s) [upload=284675]: Next returned nil segment Oct 7 00:59:14.718532982 (323h28m0.180625465s) [upload=284675]: Waiting for error group managing segments... Oct 7 00:59:14.997286224 (323h28m0.459378697s) [noise=true , address=145.79.227.5:4222 , piece_id=KOAPDQKMHV3AGUVR5K7C3XETOXVRQETK4YBKL3F3V52PPNK6ZZAA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.005665701 (323h28m0.467758194s) [noise=true , address=145.79.227.6:4257 , piece_id=2VDMEANJFQ3XQUU5KYA3K6M3KJ57JS4ECMDVWWMEBBCWPQ46HXIA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.012184233 (323h28m0.474276716s) [noise=true , address=145.79.227.14:4838 , piece_id=6WPMFZ3POXZ2O7OJSEOYI7JH77AVHA36IKZV4FBVKARJXOIUP6TA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.015080098 (323h28m0.477172581s) [noise=true , address=145.79.227.6:4258 , piece_id=KTKXNLQEOA4YMN2J67SVIEHJ7SJAIMWD2WGEJ2J2RW3ODQQBJFAQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.015967287 (323h28m0.47805976s) [noise=true , address=145.79.227.17:4969 , piece_id=34HLNC3V7GXQU4PY6GUHWJP5UNNHPRMBB2NLUAYV7EUUAWKWCPUQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.017137155 (323h28m0.479229628s) [noise=true , address=145.79.227.13:4802 , piece_id=4JLFWK3TBNSL24R7TMCTTCHTC4IJYJXZ7MNEEU53TEJNMZGCR6IQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.033318026 (323h28m0.495410509s) [noise=true , address=145.79.227.4:4185 , piece_id=NGXAZCDFBWCQVL7RMLRHRVA3QQL23JCG5YR6VM5K5GPGQWA5YL2A, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.057300663 (323h28m0.519393136s) [noise=true , address=145.79.227.19:4077 , piece_id=URF6Q57LGL3IB5CGYBSRFXA7I7MQH7TGFR2LWXPL7AJJP34SCR4Q, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.174223514 (323h28m0.636315997s) [noise=true , address=45.91.116.14:4189 , piece_id=22YZD55HNXA42ICR3R2YBA2GW7HPSEERM3DZZTXG3KOQ52IPCJWQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.247155398 (323h28m0.709247871s) [noise=true , address=45.91.116.13:4153 , piece_id=G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.247337565 (323h28m0.709430048s) [noise=true , address=77.90.31.17:4770 , piece_id=OA5BNZNICFW4ALFRXZ5UJX5KA6YQJJ5MFHJHRFX4Q23FUI3LQTQA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.314344858 (323h28m0.776437341s) [noise=true , address=77.90.31.17:4771 , piece_id=HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.323081303 (323h28m0.785173787s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y6W4Z6X6RO25JNYNRCY66VJCN5JSCDKHOJVQTDRKMKO2RWCGHVQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.323112513 (323h28m0.785204996s) [seg_pos={0 0}, upload=284675]: Segment reached optimal threshold of 13 pieces. Oct 7 00:59:15.323280172 (323h28m0.785372655s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323294269 (323h28m0.785386742s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323360255 (323h28m0.785452728s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323367799 (323h28m0.785460282s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323548955 (323h28m0.785641438s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323620661 (323h28m0.785713144s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323725931 (323h28m0.785818414s) [seg_pos={0 0}, upload=284675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:59:15.323845278 (323h28m0.785937751s) [upload=284675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:59:15.323856119 (323h28m0.785948602s) [upload=284675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:59:15.677146474 (323h28m1.139238947s) [upload=284675]: Done uploading segments... Oct 7 00:59:15.677173235 (323h28m1.139265718s) [upload=284675]: Upload finished. err: <nil> Duration: 6.210379197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................................X.XXX...X.......] duration: 6.700427431s FAIL (2025-10-07 01:00:10 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.70047349s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................................X.XXX...X......] duration: 18.12080563s FAIL (2025-10-07 00:59:52 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 18.120835577s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................................X..XX..XX...X] duration: 30.002479192s FAIL (2025-10-07 00:59:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q) context deadline exceeded; piecestore: (Node ID: 12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, Piece ID: HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ) context deadline exceeded ======LOGS====== Oct 7 00:59:12.685186317 (323h27m58.14727879s) [upload=284675]: Starting upload Oct 7 00:59:12.685278293 (323h27m58.147370776s) [upload=284675]: Done starting upload Oct 7 00:59:12.685454518 (323h27m58.147547001s) [upload=284675]: Uploading segments... Oct 7 00:59:12.685581469 (323h27m58.147673942s) [upload=284675]: Got next segment. Inline: false Oct 7 00:59:12.685608781 (323h27m58.147701265s) [upload=284675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:59:12.685613420 (323h27m58.147705903s) [upload=284675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:59:13.016521515 (323h27m58.478613998s) [seg_pos={0 0}, upload=284675]: Begin upload segment... Oct 7 00:59:13.016598392 (323h27m58.478690875s) [seg_pos={0 0}, upload=284675]: Done begin upload segment. Oct 7 00:59:13.016636374 (323h27m58.478728857s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016653387 (323h27m58.47874586s) [noise=true , address=145.79.227.19:4077 , piece_id=URF6Q57LGL3IB5CGYBSRFXA7I7MQH7TGFR2LWXPL7AJJP34SCR4Q, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016711817 (323h27m58.4788043s) [noise=true , address=145.79.227.13:4802 , piece_id=4JLFWK3TBNSL24R7TMCTTCHTC4IJYJXZ7MNEEU53TEJNMZGCR6IQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.016777923 (323h27m58.478870406s) [noise=true , address=45.91.116.14:4189 , piece_id=22YZD55HNXA42ICR3R2YBA2GW7HPSEERM3DZZTXG3KOQ52IPCJWQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017181683 (323h27m58.479274166s) [noise=true , address=145.79.227.14:4838 , piece_id=6WPMFZ3POXZ2O7OJSEOYI7JH77AVHA36IKZV4FBVKARJXOIUP6TA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017556496 (323h27m58.47964897s) [noise=true , address=145.79.227.5:4222 , piece_id=KOAPDQKMHV3AGUVR5K7C3XETOXVRQETK4YBKL3F3V52PPNK6ZZAA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.017921681 (323h27m58.480014164s) [noise=true , address=145.79.227.6:4258 , piece_id=KTKXNLQEOA4YMN2J67SVIEHJ7SJAIMWD2WGEJ2J2RW3ODQQBJFAQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.018279232 (323h27m58.480371715s) [noise=true , address=145.79.227.6:4257 , piece_id=2VDMEANJFQ3XQUU5KYA3K6M3KJ57JS4ECMDVWWMEBBCWPQ46HXIA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.018971991 (323h27m58.481064474s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.019174026 (323h27m58.481266499s) [noise=true , address=45.91.116.13:4153 , piece_id=G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.019858951 (323h27m58.481951424s) [noise=true , address=145.79.227.17:4969 , piece_id=34HLNC3V7GXQU4PY6GUHWJP5UNNHPRMBB2NLUAYV7EUUAWKWCPUQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.020800053 (323h27m58.482892536s) [noise=true , address=77.90.31.17:4771 , piece_id=HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.020990314 (323h27m58.483082807s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.022234434 (323h27m58.484326917s) [noise=true , address=77.90.31.17:4770 , piece_id=OA5BNZNICFW4ALFRXZ5UJX5KA6YQJJ5MFHJHRFX4Q23FUI3LQTQA, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.025144545 (323h27m58.487237029s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y6W4Z6X6RO25JNYNRCY66VJCN5JSCDKHOJVQTDRKMKO2RWCGHVQ, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:13.027054692 (323h27m58.489147175s) [noise=true , address=145.79.227.4:4185 , piece_id=NGXAZCDFBWCQVL7RMLRHRVA3QQL23JCG5YR6VM5K5GPGQWA5YL2A, seg_pos={0 0}, upload=284675]: Uploading piece... Oct 7 00:59:14.718509397 (323h28m0.18060188s) [upload=284675]: Next returned nil segment Oct 7 00:59:14.718532982 (323h28m0.180625465s) [upload=284675]: Waiting for error group managing segments... Oct 7 00:59:14.997286224 (323h28m0.459378697s) [noise=true , address=145.79.227.5:4222 , piece_id=KOAPDQKMHV3AGUVR5K7C3XETOXVRQETK4YBKL3F3V52PPNK6ZZAA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.005665701 (323h28m0.467758194s) [noise=true , address=145.79.227.6:4257 , piece_id=2VDMEANJFQ3XQUU5KYA3K6M3KJ57JS4ECMDVWWMEBBCWPQ46HXIA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.012184233 (323h28m0.474276716s) [noise=true , address=145.79.227.14:4838 , piece_id=6WPMFZ3POXZ2O7OJSEOYI7JH77AVHA36IKZV4FBVKARJXOIUP6TA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.015080098 (323h28m0.477172581s) [noise=true , address=145.79.227.6:4258 , piece_id=KTKXNLQEOA4YMN2J67SVIEHJ7SJAIMWD2WGEJ2J2RW3ODQQBJFAQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.015967287 (323h28m0.47805976s) [noise=true , address=145.79.227.17:4969 , piece_id=34HLNC3V7GXQU4PY6GUHWJP5UNNHPRMBB2NLUAYV7EUUAWKWCPUQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.017137155 (323h28m0.479229628s) [noise=true , address=145.79.227.13:4802 , piece_id=4JLFWK3TBNSL24R7TMCTTCHTC4IJYJXZ7MNEEU53TEJNMZGCR6IQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.033318026 (323h28m0.495410509s) [noise=true , address=145.79.227.4:4185 , piece_id=NGXAZCDFBWCQVL7RMLRHRVA3QQL23JCG5YR6VM5K5GPGQWA5YL2A, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.057300663 (323h28m0.519393136s) [noise=true , address=145.79.227.19:4077 , piece_id=URF6Q57LGL3IB5CGYBSRFXA7I7MQH7TGFR2LWXPL7AJJP34SCR4Q, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.174223514 (323h28m0.636315997s) [noise=true , address=45.91.116.14:4189 , piece_id=22YZD55HNXA42ICR3R2YBA2GW7HPSEERM3DZZTXG3KOQ52IPCJWQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.247155398 (323h28m0.709247871s) [noise=true , address=45.91.116.13:4153 , piece_id=G5YQS34BQTKA5FUV226ADJ4E7J3HGABUIAPEWWLAMNJMY6DHGR2Q, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.247337565 (323h28m0.709430048s) [noise=true , address=77.90.31.17:4770 , piece_id=OA5BNZNICFW4ALFRXZ5UJX5KA6YQJJ5MFHJHRFX4Q23FUI3LQTQA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.314344858 (323h28m0.776437341s) [noise=true , address=77.90.31.17:4771 , piece_id=HN6TMWXQYYEMH4TGKXSQ3I4WLKTFLHWNUZFEUQZ6634XEIUQWNIQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.323081303 (323h28m0.785173787s) [noise=true , address=45.91.116.14:4190 , piece_id=6Y6W4Z6X6RO25JNYNRCY66VJCN5JSCDKHOJVQTDRKMKO2RWCGHVQ, seg_pos={0 0}, upload=284675]: Done uploading piece. err: <nil> Oct 7 00:59:15.323112513 (323h28m0.785204996s) [seg_pos={0 0}, upload=284675]: Segment reached optimal threshold of 13 pieces. Oct 7 00:59:15.323280172 (323h28m0.785372655s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323294269 (323h28m0.785386742s) [noise=true , address=77.90.31.16:4734 , piece_id=P5WRLOZS5S5AYOAVOMNDXLJLPV7G64IRKDK6AVYQXDUCDRQSVI5A, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323360255 (323h28m0.785452728s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323367799 (323h28m0.785460282s) [noise=true , address=45.91.116.4:4937 , piece_id=U27VKK26QAZMH6DUESESIBNQJFI6WOVMAC4Z47EGPYACPHVNXXVA, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323548955 (323h28m0.785641438s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Oct 7 00:59:15.323620661 (323h28m0.785713144s) [noise=true , address=77.90.31.11:4555 , piece_id=6IRBYIKK5LP4IRADJC3K25QHOJ4HVMXUSNBRWPPP3PNUINOX4WNA, seg_pos={0 0}, upload=284675]: Long tail cancellation due to optimal threshold Oct 7 00:59:15.323725931 (323h28m0.785818414s) [seg_pos={0 0}, upload=284675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:59:15.323845278 (323h28m0.785937751s) [upload=284675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:59:15.323856119 (323h28m0.785948602s) [upload=284675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:59:15.677146474 (323h28m1.139238947s) [upload=284675]: Done uploading segments... Oct 7 00:59:15.677173235 (323h28m1.139265718s) [upload=284675]: Upload finished. err: <nil> Duration: 30.002560558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 28.852701972s FAIL (2025-10-07 00:59:34 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 28.852733713s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................................X.XXX...X.....] duration: 8.106485594s FAIL (2025-10-07 00:58:37 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 8.10654163s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................................X..XX..XX...] duration: 6.121925497s FAIL (2025-10-07 00:58:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA) context deadline exceeded ======LOGS====== t 7 00:56:00.189858132 (323h24m45.651950605s) [upload=284627]: Starting upload Oct 7 00:56:00.189920731 (323h24m45.652013214s) [upload=284627]: Done starting upload Oct 7 00:56:00.190097077 (323h24m45.65218956s) [upload=284627]: Uploading segments... Oct 7 00:56:00.190132274 (323h24m45.652224747s) [upload=284627]: Got next segment. Inline: false Oct 7 00:56:00.190160387 (323h24m45.65225286s) [upload=284627]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:56:00.190164555 (323h24m45.652257028s) [upload=284627]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:56:00.709885871 (323h24m46.171978354s) [seg_pos={0 0}, upload=284627]: Begin upload segment... Oct 7 00:56:00.709967937 (323h24m46.17206041s) [seg_pos={0 0}, upload=284627]: Done begin upload segment. Oct 7 00:56:00.710011539 (323h24m46.172104022s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710063549 (323h24m46.172156022s) [noise=true , address=45.91.116.5:4298 , piece_id=5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710559473 (323h24m46.172651946s) [noise=true , address=145.79.227.13:4802 , piece_id=X2RVL3C75CR46WERHD5YZPDXYSUUQCT3CNUT7ZE2JJI5HOXBE53A, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710846559 (323h24m46.172939032s) [noise=true , address=145.79.227.5:4223 , piece_id=LBA7LJD2DWQORYTPNQZRFDMN5OH7MYY73U2HKMYW6Q3A5CPE3O7A, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711109100 (323h24m46.173201573s) [noise=true , address=145.79.227.17:4972 , piece_id=AQ4SJRPFS53DSDHBQT5DJHCSOHEDHEZIV2BTXJJE62KA6NVHAUGQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711369365 (323h24m46.173461838s) [noise=true , address=45.91.116.6:4975 , piece_id=RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711696338 (323h24m46.173788811s) [noise=true , address=145.79.227.14:4839 , piece_id=3QKPNJOJMCOV5JNOBMTTLTDI5ZU3ENLVCGCYA5DXTFNWUD22SQEA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711973806 (323h24m46.174066279s) [noise=true , address=145.79.227.11:4729 , piece_id=FYDZUZ2XNEHRCFX3X3SC32OOIPJF5YR2FB5YCZGF4JXBWQ34APSA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.712251736 (323h24m46.174344209s) [noise=true , address=145.79.227.18:4006 , piece_id=UOJP37PRFGLWP6BDA6J5ZU35IL4WUN4EOTFO3K3JSG3W3PK52BIQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.714188002 (323h24m46.176280475s) [noise=true , address=45.91.116.7:4011 , piece_id=RWIDEG7LO7VG7BPF5RTRSKFVSYXWH5QQSBAL75BAIXXD7GZZ5QMQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.715636901 (323h24m46.177729384s) [noise=true , address=145.79.227.4:4185 , piece_id=SXNOEM4Y2J6VMIHXESHVBFZA2CU6IMTJM4OSWU5UASCREL5YH6WQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.719018861 (323h24m46.181111344s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.722245726 (323h24m46.184338209s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ7CYQRKOAR7N2JK533UBVFK3B5B357G3PQHATDG54VRESQD547Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.725012714 (323h24m46.187105197s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.726486110 (323h24m46.188578584s) [noise=true , address=145.79.227.6:4259 , piece_id=NSNCG3KV3IXTHAMJQPDZUETAUDJSSVHMG43YQMJAUHN644XVRUPQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.730841143 (323h24m46.192933626s) [noise=true , address=45.91.116.5:4299 , piece_id=IMQBKU7VBZLB4NZVMGFL4MYMVUTZ76NSMKYSGS63R2OAPEJ4M37Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:02.423096965 (323h24m47.885189448s) [upload=284627]: Next returned nil segment Oct 7 00:56:02.423122163 (323h24m47.885214636s) [upload=284627]: Waiting for error group managing segments... Oct 7 00:56:02.716933639 (323h24m48.179026122s) [noise=true , address=145.79.227.11:4729 , piece_id=FYDZUZ2XNEHRCFX3X3SC32OOIPJF5YR2FB5YCZGF4JXBWQ34APSA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.719843060 (323h24m48.181935543s) [noise=true , address=145.79.227.14:4839 , piece_id=3QKPNJOJMCOV5JNOBMTTLTDI5ZU3ENLVCGCYA5DXTFNWUD22SQEA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.720087635 (323h24m48.182180118s) [noise=true , address=145.79.227.17:4972 , piece_id=AQ4SJRPFS53DSDHBQT5DJHCSOHEDHEZIV2BTXJJE62KA6NVHAUGQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.727934226 (323h24m48.190026709s) [noise=true , address=145.79.227.5:4223 , piece_id=LBA7LJD2DWQORYTPNQZRFDMN5OH7MYY73U2HKMYW6Q3A5CPE3O7A, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.733271680 (323h24m48.195364153s) [noise=true , address=145.79.227.18:4006 , piece_id=UOJP37PRFGLWP6BDA6J5ZU35IL4WUN4EOTFO3K3JSG3W3PK52BIQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.733506647 (323h24m48.19559914s) [noise=true , address=145.79.227.6:4259 , piece_id=NSNCG3KV3IXTHAMJQPDZUETAUDJSSVHMG43YQMJAUHN644XVRUPQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.735797249 (323h24m48.197889742s) [noise=true , address=145.79.227.4:4185 , piece_id=SXNOEM4Y2J6VMIHXESHVBFZA2CU6IMTJM4OSWU5UASCREL5YH6WQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.743416728 (323h24m48.205509211s) [noise=true , address=145.79.227.13:4802 , piece_id=X2RVL3C75CR46WERHD5YZPDXYSUUQCT3CNUT7ZE2JJI5HOXBE53A, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.926212611 (323h24m48.388305085s) [noise=true , address=45.91.116.6:4975 , piece_id=RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.956742295 (323h24m48.418834768s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ7CYQRKOAR7N2JK533UBVFK3B5B357G3PQHATDG54VRESQD547Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.961749390 (323h24m48.423841873s) [noise=true , address=45.91.116.5:4299 , piece_id=IMQBKU7VBZLB4NZVMGFL4MYMVUTZ76NSMKYSGS63R2OAPEJ4M37Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.040720374 (323h24m48.502812857s) [noise=true , address=45.91.116.5:4298 , piece_id=5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.043108311 (323h24m48.505200784s) [noise=true , address=45.91.116.7:4011 , piece_id=RWIDEG7LO7VG7BPF5RTRSKFVSYXWH5QQSBAL75BAIXXD7GZZ5QMQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.043133209 (323h24m48.505225692s) [seg_pos={0 0}, upload=284627]: Segment reached optimal threshold of 13 pieces. Oct 7 00:56:03.043283535 (323h24m48.505376018s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 7 00:56:03.043297180 (323h24m48.505389653s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043544391 (323h24m48.505636874s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection Oct 7 00:56:03.043555633 (323h24m48.505648116s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043658399 (323h24m48.505750882s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 7 00:56:03.043667406 (323h24m48.505759889s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043684769 (323h24m48.505777252s) [seg_pos={0 0}, upload=284627]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:56:03.043766575 (323h24m48.505859058s) [upload=284627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:56:03.043773057 (323h24m48.50586554s) [upload=284627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:56:05.885462471 (323h24m51.347554964s) [upload=284627]: Done uploading segments... Oct 7 00:56:05.885488640 (323h24m51.347581124s) [upload=284627]: Upload finished. err: <nil> Duration: 6.121957428s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.986583783s FAIL (2025-10-07 00:58:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 5.986623378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.94838116s FAIL (2025-10-07 00:58:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 5.948415185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................................X.XXX...X....] duration: 5.976962933s FAIL (2025-10-07 00:57:56 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.976993792s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................X..XX..XX..] duration: 6.085000873s FAIL (2025-10-07 00:57:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA) context deadline exceeded ======LOGS====== t 7 00:56:00.189858132 (323h24m45.651950605s) [upload=284627]: Starting upload Oct 7 00:56:00.189920731 (323h24m45.652013214s) [upload=284627]: Done starting upload Oct 7 00:56:00.190097077 (323h24m45.65218956s) [upload=284627]: Uploading segments... Oct 7 00:56:00.190132274 (323h24m45.652224747s) [upload=284627]: Got next segment. Inline: false Oct 7 00:56:00.190160387 (323h24m45.65225286s) [upload=284627]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:56:00.190164555 (323h24m45.652257028s) [upload=284627]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:56:00.709885871 (323h24m46.171978354s) [seg_pos={0 0}, upload=284627]: Begin upload segment... Oct 7 00:56:00.709967937 (323h24m46.17206041s) [seg_pos={0 0}, upload=284627]: Done begin upload segment. Oct 7 00:56:00.710011539 (323h24m46.172104022s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710063549 (323h24m46.172156022s) [noise=true , address=45.91.116.5:4298 , piece_id=5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710559473 (323h24m46.172651946s) [noise=true , address=145.79.227.13:4802 , piece_id=X2RVL3C75CR46WERHD5YZPDXYSUUQCT3CNUT7ZE2JJI5HOXBE53A, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710846559 (323h24m46.172939032s) [noise=true , address=145.79.227.5:4223 , piece_id=LBA7LJD2DWQORYTPNQZRFDMN5OH7MYY73U2HKMYW6Q3A5CPE3O7A, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711109100 (323h24m46.173201573s) [noise=true , address=145.79.227.17:4972 , piece_id=AQ4SJRPFS53DSDHBQT5DJHCSOHEDHEZIV2BTXJJE62KA6NVHAUGQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711369365 (323h24m46.173461838s) [noise=true , address=45.91.116.6:4975 , piece_id=RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711696338 (323h24m46.173788811s) [noise=true , address=145.79.227.14:4839 , piece_id=3QKPNJOJMCOV5JNOBMTTLTDI5ZU3ENLVCGCYA5DXTFNWUD22SQEA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711973806 (323h24m46.174066279s) [noise=true , address=145.79.227.11:4729 , piece_id=FYDZUZ2XNEHRCFX3X3SC32OOIPJF5YR2FB5YCZGF4JXBWQ34APSA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.712251736 (323h24m46.174344209s) [noise=true , address=145.79.227.18:4006 , piece_id=UOJP37PRFGLWP6BDA6J5ZU35IL4WUN4EOTFO3K3JSG3W3PK52BIQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.714188002 (323h24m46.176280475s) [noise=true , address=45.91.116.7:4011 , piece_id=RWIDEG7LO7VG7BPF5RTRSKFVSYXWH5QQSBAL75BAIXXD7GZZ5QMQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.715636901 (323h24m46.177729384s) [noise=true , address=145.79.227.4:4185 , piece_id=SXNOEM4Y2J6VMIHXESHVBFZA2CU6IMTJM4OSWU5UASCREL5YH6WQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.719018861 (323h24m46.181111344s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.722245726 (323h24m46.184338209s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ7CYQRKOAR7N2JK533UBVFK3B5B357G3PQHATDG54VRESQD547Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.725012714 (323h24m46.187105197s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.726486110 (323h24m46.188578584s) [noise=true , address=145.79.227.6:4259 , piece_id=NSNCG3KV3IXTHAMJQPDZUETAUDJSSVHMG43YQMJAUHN644XVRUPQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.730841143 (323h24m46.192933626s) [noise=true , address=45.91.116.5:4299 , piece_id=IMQBKU7VBZLB4NZVMGFL4MYMVUTZ76NSMKYSGS63R2OAPEJ4M37Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:02.423096965 (323h24m47.885189448s) [upload=284627]: Next returned nil segment Oct 7 00:56:02.423122163 (323h24m47.885214636s) [upload=284627]: Waiting for error group managing segments... Oct 7 00:56:02.716933639 (323h24m48.179026122s) [noise=true , address=145.79.227.11:4729 , piece_id=FYDZUZ2XNEHRCFX3X3SC32OOIPJF5YR2FB5YCZGF4JXBWQ34APSA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.719843060 (323h24m48.181935543s) [noise=true , address=145.79.227.14:4839 , piece_id=3QKPNJOJMCOV5JNOBMTTLTDI5ZU3ENLVCGCYA5DXTFNWUD22SQEA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.720087635 (323h24m48.182180118s) [noise=true , address=145.79.227.17:4972 , piece_id=AQ4SJRPFS53DSDHBQT5DJHCSOHEDHEZIV2BTXJJE62KA6NVHAUGQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.727934226 (323h24m48.190026709s) [noise=true , address=145.79.227.5:4223 , piece_id=LBA7LJD2DWQORYTPNQZRFDMN5OH7MYY73U2HKMYW6Q3A5CPE3O7A, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.733271680 (323h24m48.195364153s) [noise=true , address=145.79.227.18:4006 , piece_id=UOJP37PRFGLWP6BDA6J5ZU35IL4WUN4EOTFO3K3JSG3W3PK52BIQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.733506647 (323h24m48.19559914s) [noise=true , address=145.79.227.6:4259 , piece_id=NSNCG3KV3IXTHAMJQPDZUETAUDJSSVHMG43YQMJAUHN644XVRUPQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.735797249 (323h24m48.197889742s) [noise=true , address=145.79.227.4:4185 , piece_id=SXNOEM4Y2J6VMIHXESHVBFZA2CU6IMTJM4OSWU5UASCREL5YH6WQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.743416728 (323h24m48.205509211s) [noise=true , address=145.79.227.13:4802 , piece_id=X2RVL3C75CR46WERHD5YZPDXYSUUQCT3CNUT7ZE2JJI5HOXBE53A, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.926212611 (323h24m48.388305085s) [noise=true , address=45.91.116.6:4975 , piece_id=RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.956742295 (323h24m48.418834768s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ7CYQRKOAR7N2JK533UBVFK3B5B357G3PQHATDG54VRESQD547Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.961749390 (323h24m48.423841873s) [noise=true , address=45.91.116.5:4299 , piece_id=IMQBKU7VBZLB4NZVMGFL4MYMVUTZ76NSMKYSGS63R2OAPEJ4M37Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.040720374 (323h24m48.502812857s) [noise=true , address=45.91.116.5:4298 , piece_id=5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.043108311 (323h24m48.505200784s) [noise=true , address=45.91.116.7:4011 , piece_id=RWIDEG7LO7VG7BPF5RTRSKFVSYXWH5QQSBAL75BAIXXD7GZZ5QMQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.043133209 (323h24m48.505225692s) [seg_pos={0 0}, upload=284627]: Segment reached optimal threshold of 13 pieces. Oct 7 00:56:03.043283535 (323h24m48.505376018s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 7 00:56:03.043297180 (323h24m48.505389653s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043544391 (323h24m48.505636874s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection Oct 7 00:56:03.043555633 (323h24m48.505648116s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043658399 (323h24m48.505750882s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 7 00:56:03.043667406 (323h24m48.505759889s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043684769 (323h24m48.505777252s) [seg_pos={0 0}, upload=284627]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:56:03.043766575 (323h24m48.505859058s) [upload=284627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:56:03.043773057 (323h24m48.50586554s) [upload=284627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:56:05.885462471 (323h24m51.347554964s) [upload=284627]: Done uploading segments... Oct 7 00:56:05.885488640 (323h24m51.347581124s) [upload=284627]: Upload finished. err: <nil> Duration: 6.085032894s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................................X.XXX...X...] duration: 22.785334843s FAIL (2025-10-07 00:57:14 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 22.785361934s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 8.663168778s FAIL (2025-10-07 00:56:58 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 8.663199047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 41.700099117s FAIL (2025-10-07 00:56:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 41.700120197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................X..XX..XX.] duration: 8.622531798s FAIL (2025-10-07 00:56:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA) context deadline exceeded ======LOGS====== t 7 00:56:00.189858132 (323h24m45.651950605s) [upload=284627]: Starting upload Oct 7 00:56:00.189920731 (323h24m45.652013214s) [upload=284627]: Done starting upload Oct 7 00:56:00.190097077 (323h24m45.65218956s) [upload=284627]: Uploading segments... Oct 7 00:56:00.190132274 (323h24m45.652224747s) [upload=284627]: Got next segment. Inline: false Oct 7 00:56:00.190160387 (323h24m45.65225286s) [upload=284627]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:56:00.190164555 (323h24m45.652257028s) [upload=284627]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:56:00.709885871 (323h24m46.171978354s) [seg_pos={0 0}, upload=284627]: Begin upload segment... Oct 7 00:56:00.709967937 (323h24m46.17206041s) [seg_pos={0 0}, upload=284627]: Done begin upload segment. Oct 7 00:56:00.710011539 (323h24m46.172104022s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710063549 (323h24m46.172156022s) [noise=true , address=45.91.116.5:4298 , piece_id=5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710559473 (323h24m46.172651946s) [noise=true , address=145.79.227.13:4802 , piece_id=X2RVL3C75CR46WERHD5YZPDXYSUUQCT3CNUT7ZE2JJI5HOXBE53A, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710846559 (323h24m46.172939032s) [noise=true , address=145.79.227.5:4223 , piece_id=LBA7LJD2DWQORYTPNQZRFDMN5OH7MYY73U2HKMYW6Q3A5CPE3O7A, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711109100 (323h24m46.173201573s) [noise=true , address=145.79.227.17:4972 , piece_id=AQ4SJRPFS53DSDHBQT5DJHCSOHEDHEZIV2BTXJJE62KA6NVHAUGQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711369365 (323h24m46.173461838s) [noise=true , address=45.91.116.6:4975 , piece_id=RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711696338 (323h24m46.173788811s) [noise=true , address=145.79.227.14:4839 , piece_id=3QKPNJOJMCOV5JNOBMTTLTDI5ZU3ENLVCGCYA5DXTFNWUD22SQEA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711973806 (323h24m46.174066279s) [noise=true , address=145.79.227.11:4729 , piece_id=FYDZUZ2XNEHRCFX3X3SC32OOIPJF5YR2FB5YCZGF4JXBWQ34APSA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.712251736 (323h24m46.174344209s) [noise=true , address=145.79.227.18:4006 , piece_id=UOJP37PRFGLWP6BDA6J5ZU35IL4WUN4EOTFO3K3JSG3W3PK52BIQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.714188002 (323h24m46.176280475s) [noise=true , address=45.91.116.7:4011 , piece_id=RWIDEG7LO7VG7BPF5RTRSKFVSYXWH5QQSBAL75BAIXXD7GZZ5QMQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.715636901 (323h24m46.177729384s) [noise=true , address=145.79.227.4:4185 , piece_id=SXNOEM4Y2J6VMIHXESHVBFZA2CU6IMTJM4OSWU5UASCREL5YH6WQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.719018861 (323h24m46.181111344s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.722245726 (323h24m46.184338209s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ7CYQRKOAR7N2JK533UBVFK3B5B357G3PQHATDG54VRESQD547Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.725012714 (323h24m46.187105197s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.726486110 (323h24m46.188578584s) [noise=true , address=145.79.227.6:4259 , piece_id=NSNCG3KV3IXTHAMJQPDZUETAUDJSSVHMG43YQMJAUHN644XVRUPQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.730841143 (323h24m46.192933626s) [noise=true , address=45.91.116.5:4299 , piece_id=IMQBKU7VBZLB4NZVMGFL4MYMVUTZ76NSMKYSGS63R2OAPEJ4M37Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:02.423096965 (323h24m47.885189448s) [upload=284627]: Next returned nil segment Oct 7 00:56:02.423122163 (323h24m47.885214636s) [upload=284627]: Waiting for error group managing segments... Oct 7 00:56:02.716933639 (323h24m48.179026122s) [noise=true , address=145.79.227.11:4729 , piece_id=FYDZUZ2XNEHRCFX3X3SC32OOIPJF5YR2FB5YCZGF4JXBWQ34APSA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.719843060 (323h24m48.181935543s) [noise=true , address=145.79.227.14:4839 , piece_id=3QKPNJOJMCOV5JNOBMTTLTDI5ZU3ENLVCGCYA5DXTFNWUD22SQEA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.720087635 (323h24m48.182180118s) [noise=true , address=145.79.227.17:4972 , piece_id=AQ4SJRPFS53DSDHBQT5DJHCSOHEDHEZIV2BTXJJE62KA6NVHAUGQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.727934226 (323h24m48.190026709s) [noise=true , address=145.79.227.5:4223 , piece_id=LBA7LJD2DWQORYTPNQZRFDMN5OH7MYY73U2HKMYW6Q3A5CPE3O7A, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.733271680 (323h24m48.195364153s) [noise=true , address=145.79.227.18:4006 , piece_id=UOJP37PRFGLWP6BDA6J5ZU35IL4WUN4EOTFO3K3JSG3W3PK52BIQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.733506647 (323h24m48.19559914s) [noise=true , address=145.79.227.6:4259 , piece_id=NSNCG3KV3IXTHAMJQPDZUETAUDJSSVHMG43YQMJAUHN644XVRUPQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.735797249 (323h24m48.197889742s) [noise=true , address=145.79.227.4:4185 , piece_id=SXNOEM4Y2J6VMIHXESHVBFZA2CU6IMTJM4OSWU5UASCREL5YH6WQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.743416728 (323h24m48.205509211s) [noise=true , address=145.79.227.13:4802 , piece_id=X2RVL3C75CR46WERHD5YZPDXYSUUQCT3CNUT7ZE2JJI5HOXBE53A, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.926212611 (323h24m48.388305085s) [noise=true , address=45.91.116.6:4975 , piece_id=RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.956742295 (323h24m48.418834768s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ7CYQRKOAR7N2JK533UBVFK3B5B357G3PQHATDG54VRESQD547Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.961749390 (323h24m48.423841873s) [noise=true , address=45.91.116.5:4299 , piece_id=IMQBKU7VBZLB4NZVMGFL4MYMVUTZ76NSMKYSGS63R2OAPEJ4M37Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.040720374 (323h24m48.502812857s) [noise=true , address=45.91.116.5:4298 , piece_id=5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.043108311 (323h24m48.505200784s) [noise=true , address=45.91.116.7:4011 , piece_id=RWIDEG7LO7VG7BPF5RTRSKFVSYXWH5QQSBAL75BAIXXD7GZZ5QMQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.043133209 (323h24m48.505225692s) [seg_pos={0 0}, upload=284627]: Segment reached optimal threshold of 13 pieces. Oct 7 00:56:03.043283535 (323h24m48.505376018s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 7 00:56:03.043297180 (323h24m48.505389653s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043544391 (323h24m48.505636874s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection Oct 7 00:56:03.043555633 (323h24m48.505648116s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043658399 (323h24m48.505750882s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 7 00:56:03.043667406 (323h24m48.505759889s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043684769 (323h24m48.505777252s) [seg_pos={0 0}, upload=284627]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:56:03.043766575 (323h24m48.505859058s) [upload=284627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:56:03.043773057 (323h24m48.50586554s) [upload=284627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:56:05.885462471 (323h24m51.347554964s) [upload=284627]: Done uploading segments... Oct 7 00:56:05.885488640 (323h24m51.347581124s) [upload=284627]: Upload finished. err: <nil> Duration: 8.622562277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................................X..XX..XX] duration: 30.00276922s FAIL (2025-10-07 00:56:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA) context deadline exceeded ======LOGS====== t 7 00:56:00.189858132 (323h24m45.651950605s) [upload=284627]: Starting upload Oct 7 00:56:00.189920731 (323h24m45.652013214s) [upload=284627]: Done starting upload Oct 7 00:56:00.190097077 (323h24m45.65218956s) [upload=284627]: Uploading segments... Oct 7 00:56:00.190132274 (323h24m45.652224747s) [upload=284627]: Got next segment. Inline: false Oct 7 00:56:00.190160387 (323h24m45.65225286s) [upload=284627]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:56:00.190164555 (323h24m45.652257028s) [upload=284627]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:56:00.709885871 (323h24m46.171978354s) [seg_pos={0 0}, upload=284627]: Begin upload segment... Oct 7 00:56:00.709967937 (323h24m46.17206041s) [seg_pos={0 0}, upload=284627]: Done begin upload segment. Oct 7 00:56:00.710011539 (323h24m46.172104022s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710063549 (323h24m46.172156022s) [noise=true , address=45.91.116.5:4298 , piece_id=5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710559473 (323h24m46.172651946s) [noise=true , address=145.79.227.13:4802 , piece_id=X2RVL3C75CR46WERHD5YZPDXYSUUQCT3CNUT7ZE2JJI5HOXBE53A, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.710846559 (323h24m46.172939032s) [noise=true , address=145.79.227.5:4223 , piece_id=LBA7LJD2DWQORYTPNQZRFDMN5OH7MYY73U2HKMYW6Q3A5CPE3O7A, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711109100 (323h24m46.173201573s) [noise=true , address=145.79.227.17:4972 , piece_id=AQ4SJRPFS53DSDHBQT5DJHCSOHEDHEZIV2BTXJJE62KA6NVHAUGQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711369365 (323h24m46.173461838s) [noise=true , address=45.91.116.6:4975 , piece_id=RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711696338 (323h24m46.173788811s) [noise=true , address=145.79.227.14:4839 , piece_id=3QKPNJOJMCOV5JNOBMTTLTDI5ZU3ENLVCGCYA5DXTFNWUD22SQEA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.711973806 (323h24m46.174066279s) [noise=true , address=145.79.227.11:4729 , piece_id=FYDZUZ2XNEHRCFX3X3SC32OOIPJF5YR2FB5YCZGF4JXBWQ34APSA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.712251736 (323h24m46.174344209s) [noise=true , address=145.79.227.18:4006 , piece_id=UOJP37PRFGLWP6BDA6J5ZU35IL4WUN4EOTFO3K3JSG3W3PK52BIQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.714188002 (323h24m46.176280475s) [noise=true , address=45.91.116.7:4011 , piece_id=RWIDEG7LO7VG7BPF5RTRSKFVSYXWH5QQSBAL75BAIXXD7GZZ5QMQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.715636901 (323h24m46.177729384s) [noise=true , address=145.79.227.4:4185 , piece_id=SXNOEM4Y2J6VMIHXESHVBFZA2CU6IMTJM4OSWU5UASCREL5YH6WQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.719018861 (323h24m46.181111344s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.722245726 (323h24m46.184338209s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ7CYQRKOAR7N2JK533UBVFK3B5B357G3PQHATDG54VRESQD547Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.725012714 (323h24m46.187105197s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.726486110 (323h24m46.188578584s) [noise=true , address=145.79.227.6:4259 , piece_id=NSNCG3KV3IXTHAMJQPDZUETAUDJSSVHMG43YQMJAUHN644XVRUPQ, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:00.730841143 (323h24m46.192933626s) [noise=true , address=45.91.116.5:4299 , piece_id=IMQBKU7VBZLB4NZVMGFL4MYMVUTZ76NSMKYSGS63R2OAPEJ4M37Q, seg_pos={0 0}, upload=284627]: Uploading piece... Oct 7 00:56:02.423096965 (323h24m47.885189448s) [upload=284627]: Next returned nil segment Oct 7 00:56:02.423122163 (323h24m47.885214636s) [upload=284627]: Waiting for error group managing segments... Oct 7 00:56:02.716933639 (323h24m48.179026122s) [noise=true , address=145.79.227.11:4729 , piece_id=FYDZUZ2XNEHRCFX3X3SC32OOIPJF5YR2FB5YCZGF4JXBWQ34APSA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.719843060 (323h24m48.181935543s) [noise=true , address=145.79.227.14:4839 , piece_id=3QKPNJOJMCOV5JNOBMTTLTDI5ZU3ENLVCGCYA5DXTFNWUD22SQEA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.720087635 (323h24m48.182180118s) [noise=true , address=145.79.227.17:4972 , piece_id=AQ4SJRPFS53DSDHBQT5DJHCSOHEDHEZIV2BTXJJE62KA6NVHAUGQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.727934226 (323h24m48.190026709s) [noise=true , address=145.79.227.5:4223 , piece_id=LBA7LJD2DWQORYTPNQZRFDMN5OH7MYY73U2HKMYW6Q3A5CPE3O7A, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.733271680 (323h24m48.195364153s) [noise=true , address=145.79.227.18:4006 , piece_id=UOJP37PRFGLWP6BDA6J5ZU35IL4WUN4EOTFO3K3JSG3W3PK52BIQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.733506647 (323h24m48.19559914s) [noise=true , address=145.79.227.6:4259 , piece_id=NSNCG3KV3IXTHAMJQPDZUETAUDJSSVHMG43YQMJAUHN644XVRUPQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.735797249 (323h24m48.197889742s) [noise=true , address=145.79.227.4:4185 , piece_id=SXNOEM4Y2J6VMIHXESHVBFZA2CU6IMTJM4OSWU5UASCREL5YH6WQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.743416728 (323h24m48.205509211s) [noise=true , address=145.79.227.13:4802 , piece_id=X2RVL3C75CR46WERHD5YZPDXYSUUQCT3CNUT7ZE2JJI5HOXBE53A, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.926212611 (323h24m48.388305085s) [noise=true , address=45.91.116.6:4975 , piece_id=RTDSQJSMJ27Y7FDPWNCL737W7536F5Q23IAWQV7FRDVCL2B6K5PA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.956742295 (323h24m48.418834768s) [noise=true , address=45.91.116.4:4938 , piece_id=MQ7CYQRKOAR7N2JK533UBVFK3B5B357G3PQHATDG54VRESQD547Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:02.961749390 (323h24m48.423841873s) [noise=true , address=45.91.116.5:4299 , piece_id=IMQBKU7VBZLB4NZVMGFL4MYMVUTZ76NSMKYSGS63R2OAPEJ4M37Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.040720374 (323h24m48.502812857s) [noise=true , address=45.91.116.5:4298 , piece_id=5MSY3KV6R5N7SO222WTGJRDSX2FB5R7TPLLZELWNXJMI2BU5KYTA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.043108311 (323h24m48.505200784s) [noise=true , address=45.91.116.7:4011 , piece_id=RWIDEG7LO7VG7BPF5RTRSKFVSYXWH5QQSBAL75BAIXXD7GZZ5QMQ, seg_pos={0 0}, upload=284627]: Done uploading piece. err: <nil> Oct 7 00:56:03.043133209 (323h24m48.505225692s) [seg_pos={0 0}, upload=284627]: Segment reached optimal threshold of 13 pieces. Oct 7 00:56:03.043283535 (323h24m48.505376018s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: expected piece hash; context canceled; EOF Oct 7 00:56:03.043297180 (323h24m48.505389653s) [noise=true , address=77.90.31.16:4733 , piece_id=MNSWXFFMHCVXPZWJFCMODNOFNXK7M22QFPLNZXUMJ6FZVDFCCTVA, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043544391 (323h24m48.505636874s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40340->45.91.116.8:4048: use of closed network connection Oct 7 00:56:03.043555633 (323h24m48.505648116s) [noise=true , address=45.91.116.8:4048 , piece_id=7FDZCYNIFWKSA4OG4WIJKND4MIWGBLC4HHGG3ISSHRCTK7Y7NN2Q, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043658399 (323h24m48.505750882s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 7 00:56:03.043667406 (323h24m48.505759889s) [noise=true , address=45.91.116.7:4010 , piece_id=MSDT6Y5T4ZUKYZWJFOXPXOKOD7NYCKLZFK2KO2DLEVPGFBNORLDA, seg_pos={0 0}, upload=284627]: Long tail cancellation due to optimal threshold Oct 7 00:56:03.043684769 (323h24m48.505777252s) [seg_pos={0 0}, upload=284627]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:56:03.043766575 (323h24m48.505859058s) [upload=284627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:56:03.043773057 (323h24m48.50586554s) [upload=284627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:56:05.885462471 (323h24m51.347554964s) [upload=284627]: Done uploading segments... Oct 7 00:56:05.885488640 (323h24m51.347581124s) [upload=284627]: Upload finished. err: <nil> Duration: 30.00280091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................X.XXX...X..] duration: 9.820171428s FAIL (2025-10-07 00:56:24 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 9.820212476s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................X.XXX...X.] duration: 13.793026674s FAIL (2025-10-07 00:56:12 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 13.793062011s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 54.962443807s FAIL (2025-10-07 00:56:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 54.962499072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................X..XX..X] duration: 30.004623822s FAIL (2025-10-07 00:55:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, Piece ID: RWJVWB7E7MJ5CGWRJVEPT2W6DXMHKNC7ZUGI3TPX3Z4CWCMCU3NA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: YNSPDFL6N4GP3FBO2B35Q52TUTMYDSX4Y6QX2FP3MAFBENBWJ4NQ) context deadline exceeded ======LOGS====== 7 00:55:31.364600473 (323h24m16.826692946s) [upload=284616]: Starting upload Oct 7 00:55:31.364676367 (323h24m16.826768851s) [upload=284616]: Done starting upload Oct 7 00:55:31.364761981 (323h24m16.826854464s) [upload=284616]: Uploading segments... Oct 7 00:55:31.364871610 (323h24m16.826964083s) [upload=284616]: Got next segment. Inline: false Oct 7 00:55:31.364903090 (323h24m16.826995573s) [upload=284616]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:55:31.364907698 (323h24m16.827000171s) [upload=284616]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:55:36.869557117 (323h24m22.33164959s) [seg_pos={0 0}, upload=284616]: Begin upload segment... Oct 7 00:55:36.869617893 (323h24m22.331710366s) [seg_pos={0 0}, upload=284616]: Done begin upload segment. Oct 7 00:55:36.869643541 (323h24m22.331736025s) [noise=true , address=145.79.227.5:4222 , piece_id=PGKXVE4H4FDNIUMKEFHAT3F6RSVPB7IHP6H3MWUTFDJJ3LMOE6UQ, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.869714326 (323h24m22.331806809s) [noise=true , address=77.90.31.15:4699 , piece_id=RWJVWB7E7MJ5CGWRJVEPT2W6DXMHKNC7ZUGI3TPX3Z4CWCMCU3NA, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.870277840 (323h24m22.332370313s) [noise=true , address=45.91.116.6:4975 , piece_id=BRM63U7FDWOU5FMRBKAEMYJTO4EIN2HVIJWCDNIEOBHUM7UAA4NA, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.870561570 (323h24m22.332654043s) [noise=true , address=45.91.116.13:4155 , piece_id=YNSPDFL6N4GP3FBO2B35Q52TUTMYDSX4Y6QX2FP3MAFBENBWJ4NQ, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.871067323 (323h24m22.333159796s) [noise=true , address=145.79.227.5:4223 , piece_id=2KU7ECYWWMI6FVVWOYZKHPLFYFZTTDJHDGQK4C7C7Y6X5B37PNKQ, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.871195146 (323h24m22.333287619s) [noise=true , address=145.79.227.2:4113 , piece_id=DPADX3A477HSYVPJCGRDYW2Z4NGGSZRZQJ2PIP3GSRDHOVZPSJ5Q, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.871376361 (323h24m22.333468834s) [noise=true , address=45.91.116.8:4048 , piece_id=AAK2GJ4SOBD3X5C3JF4QP6R7PMEUSP7R2SUHPCAR2FWZ6PB7KILA, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.871627329 (323h24m22.333719802s) [noise=true , address=77.90.31.12:4592 , piece_id=XWQUEGFIPBQARRPXR2C5UA222QAV6NUCVY3H6AWQOC2U3JDDQJAQ, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.871727460 (323h24m22.333819933s) [noise=true , address=145.79.227.6:4258 , piece_id=ATDDUVTRLAYS4IEB7ZIE55I66KVKTZCIEKM2MFX2PS4YMEHGA6NA, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.871987365 (323h24m22.334079858s) [noise=true , address=45.91.116.9:4263 , piece_id=Z33CEBUCICRQUGX5N77OHS2MGLUO2LERFNL6G2OFGKMOWVVOM6BA, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.872356858 (323h24m22.334449331s) [noise=true , address=145.79.227.15:4876 , piece_id=VQCZC3NCZY3FUDQTBXHVY4ZNTQVSRKSN2LVC7T2XMGVVZA2RTM4A, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.872370405 (323h24m22.334462878s) [noise=true , address=145.79.227.13:4801 , piece_id=AL37X5L75JCAHH6ZJQ43TGTSOHNOE4SINLKVXLK563ITSI4IUUJQ, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.872765577 (323h24m22.33485805s) [noise=true , address=45.91.116.14:4191 , piece_id=PC55RZSR5JK7SQ3YRZOIWTCX2EIAKIWTPS56MKXAV3PNNG3OGKLA, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.873600366 (323h24m22.335692839s) [noise=true , address=45.91.116.5:4299 , piece_id=6UMZHDW2QPYCC4GLKCCXVUYJUIRPW3FF6ZV7LBA3JQPZ5S53S6SQ, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.874835769 (323h24m22.336928242s) [noise=true , address=45.91.116.6:4973 , piece_id=7Q2MQXQLQKXCFBEHB6RDAD6WJQHLXIAQPNGCCQJOCQDWLM7QTXTQ, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:36.876134913 (323h24m22.338227396s) [noise=true , address=145.79.227.18:4006 , piece_id=Y3EF243SEPRFHWFCBIXVQPB4EKMAMG7YHYDHW7XNUH5IQLOPPK6Q, seg_pos={0 0}, upload=284616]: Uploading piece... Oct 7 00:55:38.571756612 (323h24m24.033849095s) [upload=284616]: Next returned nil segment Oct 7 00:55:38.571780387 (323h24m24.03387286s) [upload=284616]: Waiting for error group managing segments... Oct 7 00:55:38.854029317 (323h24m24.31612179s) [noise=true , address=145.79.227.18:4006 , piece_id=Y3EF243SEPRFHWFCBIXVQPB4EKMAMG7YHYDHW7XNUH5IQLOPPK6Q, seg_pos={0 0}, upload=284616]: Done uploading piece. err: <nil> Oct 7 00:55:38.860406900 (323h24m24.322499383s) [noise=true , address=145.79.227.2:4113 , piece_id=DPADX3A477HSYVPJCGRDYW2Z4NGGSZRZQJ2PIP3GSRDHOVZPSJ5Q, seg_pos={0 0}, upload=284616]: Done uploading piece. err: <nil> Oct 7 00:55:38.861701606 (323h24m24.323794079s) [noise=true , address=145.79.227.15:4876 , piece_id=VQCZC3NCZY3FUDQTBXHVY4ZNTQVSRKSN2LVC7T2XMGVVZA2RTM4A, seg_pos={0 0}, upload=284616]: Done uploading piece. err: <nil> Oct 7 00:55:38.866180034 (323h24m24.328272517s) [noise=true , address=145.79.227.13:4801 , piece_id=AL37X5L75JCAHH6ZJQ43TGTSOHNOE4SINLKVXLK563ITSI4IUUJQ, seg_pos={0 0}, upload=284616]: Done uploading piece. err: <nil> Oct 7 00:55:38.876249858 (323h24m24.338342331s) [noise=true , address=145.79.227.5:4223 , piece_id=2KU7ECYWWMI6FVVWOYZKHPLFYFZTTDJHDGQK4C7C7Y6X5B37PNKQ, seg_pos={0 0}, upload=284616]: Done uploading piece. err: <nil> Oct 7 00:55:38.881500156 (323h24m24.343592629s) [noise=true , address=145.79.227.5:4222 , piece_id=PGKXVE4H4FDNIUMKEFHAT3F6RSVPB7IHP6H3MWUTFDJJ3LMOE6UQ, seg_pos={0 0}, upload=284616]: Done uploading piece. err: <nil> Oct 7 00:55:38.881630093 (323h24m24.343722576s) [noise=true , address=145.79.227.6:4258 , piece_id=ATDDUVTRLAYS4IEB7ZIE55I66KVKTZCIEKM2MFX2PS4YMEHGA6NA, seg_pos={0 0}, upload=284616]: Done uploading piece. err: <nil> Oct 7 00:55:39.019887148 (323h24m24.481979621s) [noise=true , address=45.91.116.6:4973 , piece_id=7Q2MQXQLQKXCFBEHB6RDAD6WJQHLXIAQPNGCCQJOCQDWLM7QTXTQ, seg_pos={0 0}, upload=284616]: Done uploading piece. err: <nil> Oct 7 00:55:39.053349496 (323h24m24.515441969s) [noise=true , address=45.91.116.5:4299 , piece_id=6UMZHDW2QPYCC4GLKCCXVUYJUIRPW3FF6ZV7LBA3JQPZ5S53S6SQ, seg_pos={0 0}, upload=284616]: Done uploading piece. err: <nil> Oct 7 00:55:39.140927930 (323h24m24.603020413s) [noise=true , address=45.91.116.6:4975 , piece_id=BRM63U7FDWOU5FMRBKAEMYJTO4EIN2HVIJWCDNIEOBHUM7UAA4NA, seg_pos={0 0}, upload=284616]: Done uploading piece. err: <nil> Oct 7 00:55:39.159587500 (323h24m24.621679973s) [noise=true , address=77.90.31.12:4592 , piece_id=XWQUEGFIPBQARRPXR2C5UA222QAV6NUCVY3H6AWQOC2U3JDDQJAQ, seg_pos={0 0}, upload=284616]: Done uploading piece. err: <nil> Oct 7 00:55:39.178662933 (323h24m24.640755416s) [noise=true , address=45.91.116.13:4155 , piece_id=YNSPDFL6N4GP3FBO2B35Q52TUTMYDSX4Y6QX2FP3MAFBENBWJ4NQ, seg_pos={0 0}, upload=284616]: Done uploading piece. err: <nil> Oct 7 00:55:39.178721686 (323h24m24.640814159s) [noise=true , address=77.90.31.15:4699 , piece_id=RWJVWB7E7MJ5CGWRJVEPT2W6DXMHKNC7ZUGI3TPX3Z4CWCMCU3NA, seg_pos={0 0}, upload=284616]: Done uploading piece. err: <nil> Oct 7 00:55:39.178743818 (323h24m24.640836291s) [seg_pos={0 0}, upload=284616]: Segment reached optimal threshold of 13 pieces. Oct 7 00:55:39.178886339 (323h24m24.640978812s) [noise=true , address=45.91.116.8:4048 , piece_id=AAK2GJ4SOBD3X5C3JF4QP6R7PMEUSP7R2SUHPCAR2FWZ6PB7KILA, seg_pos={0 0}, upload=284616]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 7 00:55:39.178896058 (323h24m24.640988541s) [noise=true , address=45.91.116.8:4048 , piece_id=AAK2GJ4SOBD3X5C3JF4QP6R7PMEUSP7R2SUHPCAR2FWZ6PB7KILA, seg_pos={0 0}, upload=284616]: Long tail cancellation due to optimal threshold Oct 7 00:55:39.178919753 (323h24m24.641012226s) [noise=true , address=45.91.116.14:4191 , piece_id=PC55RZSR5JK7SQ3YRZOIWTCX2EIAKIWTPS56MKXAV3PNNG3OGKLA, seg_pos={0 0}, upload=284616]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Oct 7 00:55:39.178929912 (323h24m24.641022395s) [noise=true , address=45.91.116.14:4191 , piece_id=PC55RZSR5JK7SQ3YRZOIWTCX2EIAKIWTPS56MKXAV3PNNG3OGKLA, seg_pos={0 0}, upload=284616]: Long tail cancellation due to optimal threshold Oct 7 00:55:39.178997471 (323h24m24.641089954s) [noise=true , address=45.91.116.9:4263 , piece_id=Z33CEBUCICRQUGX5N77OHS2MGLUO2LERFNL6G2OFGKMOWVVOM6BA, seg_pos={0 0}, upload=284616]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:34934->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:34934->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34934->45.91.116.9:4263: use of closed network connection Oct 7 00:55:39.179004173 (323h24m24.641096646s) [noise=true , address=45.91.116.9:4263 , piece_id=Z33CEBUCICRQUGX5N77OHS2MGLUO2LERFNL6G2OFGKMOWVVOM6BA, seg_pos={0 0}, upload=284616]: Long tail cancellation due to optimal threshold Oct 7 00:55:39.179039611 (323h24m24.641132084s) [seg_pos={0 0}, upload=284616]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:55:39.179120605 (323h24m24.641213078s) [upload=284616]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:55:39.179124904 (323h24m24.641217377s) [upload=284616]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:55:42.447187865 (323h24m27.909280348s) [upload=284616]: Done uploading segments... Oct 7 00:55:42.447206760 (323h24m27.909299243s) [upload=284616]: Upload finished. err: <nil> Duration: 30.004655022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................................X.XXX...X] duration: 30.00079523s FAIL (2025-10-07 00:55:36 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000823966s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X..XX..] duration: 7.295046275s FAIL (2025-10-07 00:54:41 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Oct 7 00:52:22.589363902 (323h21m8.051456385s) [upload=284576]: Starting upload Oct 7 00:52:22.589458121 (323h21m8.051550594s) [upload=284576]: Done starting upload Oct 7 00:52:22.589588860 (323h21m8.051681363s) [upload=284576]: Uploading segments... Oct 7 00:52:22.589745138 (323h21m8.051837631s) [upload=284576]: Got next segment. Inline: false Oct 7 00:52:22.589791246 (323h21m8.051883729s) [upload=284576]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:52:22.589818087 (323h21m8.05191057s) [upload=284576]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:52:27.006217443 (323h21m12.468309956s) [seg_pos={0 0}, upload=284576]: Begin upload segment... Oct 7 00:52:27.006331671 (323h21m12.468424154s) [seg_pos={0 0}, upload=284576]: Done begin upload segment. Oct 7 00:52:27.006375454 (323h21m12.468467927s) [noise=true , address=145.79.227.15:4874 , piece_id=5DQFDNCETE743IDGWXSOXWUO3U42RXHRIZ42U2LSBNYFYJ4F73CQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.007934914 (323h21m12.470027387s) [noise=true , address=145.79.227.19:4079 , piece_id=LA6H5KWT45SDLUVXQLYDC2APAYTYIVLNECJEFKZEL3GKIR4RF7CQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.008623895 (323h21m12.470716368s) [noise=true , address=77.90.31.15:4697 , piece_id=S5FQAZPXXKGIZX37YAMAXMIKFYBNBGLHYIK7353NTYOIYXDUIZSA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.008964324 (323h21m12.471056797s) [noise=true , address=145.79.227.3:4151 , piece_id=CRTQEPDR6WJKBMQZP72CWE65I4ROVJBHFPWAQHPO4NU2IZ6ILNVA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.009411014 (323h21m12.471503497s) [noise=true , address=45.91.116.7:4009 , piece_id=MPTEURKI6WGHWZVTUZJACJQ646PBNZOM3QUWGBTAYRIS75JCESCQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.010011507 (323h21m12.472103991s) [noise=true , address=45.91.116.5:4298 , piece_id=S6FKHQXQOKHGLEKLNWTKSFDQURO3OWB27FQIAPCMMNHC4V3WLADQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.011233614 (323h21m12.473326098s) [noise=true , address=145.79.227.3:4150 , piece_id=OOXPQHWNAKK52AKWDEM7GEH3PPFN2FS5HSBZB7WJZ7VH365DQN6A, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.014548516 (323h21m12.476640999s) [noise=true , address=45.91.116.11:4117 , piece_id=QJIQT6PZ64YCS4MOQCSX5FY3PR4Z2OWACEJJVWXOPB5IV2DJGMVA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.017671924 (323h21m12.479764407s) [noise=true , address=145.79.227.18:4006 , piece_id=5KTVQV4YJ2V3LB7O5V2S5TIFHXBJ5NVJR244MC4KCVK42UPQOGQQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.022176361 (323h21m12.484268834s) [noise=true , address=145.79.227.16:4912 , piece_id=PPVFKSZMR6IU7RIEFAHDUNL453UXSWWCM7G7LP2OGNUEF6QP4HDA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.023896196 (323h21m12.48598867s) [noise=true , address=145.79.227.2:4113 , piece_id=QGNM2PVLQBQZLIJXHMLZMAFEFGVISFRME2DBXHGFXXVRRSFE4IBQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.026829021 (323h21m12.488921494s) [noise=true , address=77.90.31.10:4520 , piece_id=LCZKCLLUDLASEQSFIIWC4QVY3V2GH4BO4LIXKZL6KJEML76QFRXA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.027439743 (323h21m12.489532216s) [noise=true , address=145.79.227.6:4259 , piece_id=D7J5H6YIEN4VIUVYL5QQCIVVRKUH6KCOCUX3OSYV6BIHUPBR3UEQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.029282001 (323h21m12.491374484s) [noise=true , address=45.91.116.8:4048 , piece_id=OKLLUPVCU5VHCIV34L2UJMPUXOB3YZOPR6VJKIQBOF4UAK7P3M2Q, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.030439296 (323h21m12.492531769s) [noise=true , address=77.90.31.13:4628 , piece_id=K23CB3Z57UYPF4UZDQ3JN47LL4PORWUQCFPEPBS5BRFEKADOMB5A, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.032398566 (323h21m12.494491039s) [noise=true , address=145.79.227.14:4840 , piece_id=NP32LN2KO3Q2VT4WGYY53CH6TWHPILFNDJASKUFILTNSULLQNPLQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:28.730821790 (323h21m14.192914273s) [upload=284576]: Next returned nil segment Oct 7 00:52:28.730842921 (323h21m14.192935394s) [upload=284576]: Waiting for error group managing segments... Oct 7 00:52:29.018765363 (323h21m14.480857846s) [noise=true , address=145.79.227.16:4912 , piece_id=PPVFKSZMR6IU7RIEFAHDUNL453UXSWWCM7G7LP2OGNUEF6QP4HDA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.019386345 (323h21m14.481478818s) [noise=true , address=145.79.227.18:4006 , piece_id=5KTVQV4YJ2V3LB7O5V2S5TIFHXBJ5NVJR244MC4KCVK42UPQOGQQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.023241657 (323h21m14.48533414s) [noise=true , address=145.79.227.6:4259 , piece_id=D7J5H6YIEN4VIUVYL5QQCIVVRKUH6KCOCUX3OSYV6BIHUPBR3UEQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.023572917 (323h21m14.48566539s) [noise=true , address=145.79.227.15:4874 , piece_id=5DQFDNCETE743IDGWXSOXWUO3U42RXHRIZ42U2LSBNYFYJ4F73CQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.033328714 (323h21m14.495421187s) [noise=true , address=145.79.227.19:4079 , piece_id=LA6H5KWT45SDLUVXQLYDC2APAYTYIVLNECJEFKZEL3GKIR4RF7CQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.036569455 (323h21m14.498661938s) [noise=true , address=145.79.227.3:4151 , piece_id=CRTQEPDR6WJKBMQZP72CWE65I4ROVJBHFPWAQHPO4NU2IZ6ILNVA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.036935462 (323h21m14.499027945s) [noise=true , address=145.79.227.2:4113 , piece_id=QGNM2PVLQBQZLIJXHMLZMAFEFGVISFRME2DBXHGFXXVRRSFE4IBQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.069794942 (323h21m14.531887415s) [noise=true , address=145.79.227.14:4840 , piece_id=NP32LN2KO3Q2VT4WGYY53CH6TWHPILFNDJASKUFILTNSULLQNPLQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.097526738 (323h21m14.559619211s) [noise=true , address=145.79.227.3:4150 , piece_id=OOXPQHWNAKK52AKWDEM7GEH3PPFN2FS5HSBZB7WJZ7VH365DQN6A, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.151291235 (323h21m14.613383718s) [noise=true , address=45.91.116.5:4298 , piece_id=S6FKHQXQOKHGLEKLNWTKSFDQURO3OWB27FQIAPCMMNHC4V3WLADQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.226239077 (323h21m14.68833155s) [noise=true , address=45.91.116.8:4048 , piece_id=OKLLUPVCU5VHCIV34L2UJMPUXOB3YZOPR6VJKIQBOF4UAK7P3M2Q, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.235900194 (323h21m14.697992667s) [noise=true , address=77.90.31.10:4520 , piece_id=LCZKCLLUDLASEQSFIIWC4QVY3V2GH4BO4LIXKZL6KJEML76QFRXA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.249112243 (323h21m14.711204716s) [noise=true , address=77.90.31.15:4697 , piece_id=S5FQAZPXXKGIZX37YAMAXMIKFYBNBGLHYIK7353NTYOIYXDUIZSA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.249144654 (323h21m14.711237128s) [seg_pos={0 0}, upload=284576]: Segment reached optimal threshold of 13 pieces. Oct 7 00:52:29.249279160 (323h21m14.711371643s) [noise=true , address=45.91.116.11:4117 , piece_id=QJIQT6PZ64YCS4MOQCSX5FY3PR4Z2OWACEJJVWXOPB5IV2DJGMVA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Oct 7 00:52:29.249287907 (323h21m14.71138038s) [noise=true , address=45.91.116.11:4117 , piece_id=QJIQT6PZ64YCS4MOQCSX5FY3PR4Z2OWACEJJVWXOPB5IV2DJGMVA, seg_pos={0 0}, upload=284576]: Long tail cancellation due to optimal threshold Oct 7 00:52:29.249362249 (323h21m14.711454722s) [noise=true , address=45.91.116.7:4009 , piece_id=MPTEURKI6WGHWZVTUZJACJQ646PBNZOM3QUWGBTAYRIS75JCESCQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Oct 7 00:52:29.249367709 (323h21m14.711460182s) [noise=true , address=45.91.116.7:4009 , piece_id=MPTEURKI6WGHWZVTUZJACJQ646PBNZOM3QUWGBTAYRIS75JCESCQ, seg_pos={0 0}, upload=284576]: Long tail cancellation due to optimal threshold Oct 7 00:52:29.249437041 (323h21m14.711529524s) [noise=true , address=77.90.31.13:4628 , piece_id=K23CB3Z57UYPF4UZDQ3JN47LL4PORWUQCFPEPBS5BRFEKADOMB5A, seg_pos={0 0}, upload=284576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Oct 7 00:52:29.249474352 (323h21m14.711566825s) [noise=true , address=77.90.31.13:4628 , piece_id=K23CB3Z57UYPF4UZDQ3JN47LL4PORWUQCFPEPBS5BRFEKADOMB5A, seg_pos={0 0}, upload=284576]: Long tail cancellation due to optimal threshold Oct 7 00:52:29.249504179 (323h21m14.711596652s) [seg_pos={0 0}, upload=284576]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:52:29.249600092 (323h21m14.711692565s) [upload=284576]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:52:29.249606494 (323h21m14.711698967s) [upload=284576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:52:29.613316298 (323h21m15.075408771s) [upload=284576]: Done uploading segments... Oct 7 00:52:29.613340905 (323h21m15.075433378s) [upload=284576]: Upload finished. err: <nil> Duration: 7.295079138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................X.XXX...] duration: 6.661481942s FAIL (2025-10-07 00:54:37 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.661507791s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.870470985s FAIL (2025-10-07 00:54:31 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 5.870501623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................X.XXX..] duration: 6.097486621s FAIL (2025-10-07 00:54:13 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.097519001s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.684906387s FAIL (2025-10-07 00:53:36 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 5.684943478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................X..XX.] duration: 6.167060249s FAIL (2025-10-07 00:53:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Oct 7 00:52:22.589363902 (323h21m8.051456385s) [upload=284576]: Starting upload Oct 7 00:52:22.589458121 (323h21m8.051550594s) [upload=284576]: Done starting upload Oct 7 00:52:22.589588860 (323h21m8.051681363s) [upload=284576]: Uploading segments... Oct 7 00:52:22.589745138 (323h21m8.051837631s) [upload=284576]: Got next segment. Inline: false Oct 7 00:52:22.589791246 (323h21m8.051883729s) [upload=284576]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:52:22.589818087 (323h21m8.05191057s) [upload=284576]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:52:27.006217443 (323h21m12.468309956s) [seg_pos={0 0}, upload=284576]: Begin upload segment... Oct 7 00:52:27.006331671 (323h21m12.468424154s) [seg_pos={0 0}, upload=284576]: Done begin upload segment. Oct 7 00:52:27.006375454 (323h21m12.468467927s) [noise=true , address=145.79.227.15:4874 , piece_id=5DQFDNCETE743IDGWXSOXWUO3U42RXHRIZ42U2LSBNYFYJ4F73CQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.007934914 (323h21m12.470027387s) [noise=true , address=145.79.227.19:4079 , piece_id=LA6H5KWT45SDLUVXQLYDC2APAYTYIVLNECJEFKZEL3GKIR4RF7CQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.008623895 (323h21m12.470716368s) [noise=true , address=77.90.31.15:4697 , piece_id=S5FQAZPXXKGIZX37YAMAXMIKFYBNBGLHYIK7353NTYOIYXDUIZSA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.008964324 (323h21m12.471056797s) [noise=true , address=145.79.227.3:4151 , piece_id=CRTQEPDR6WJKBMQZP72CWE65I4ROVJBHFPWAQHPO4NU2IZ6ILNVA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.009411014 (323h21m12.471503497s) [noise=true , address=45.91.116.7:4009 , piece_id=MPTEURKI6WGHWZVTUZJACJQ646PBNZOM3QUWGBTAYRIS75JCESCQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.010011507 (323h21m12.472103991s) [noise=true , address=45.91.116.5:4298 , piece_id=S6FKHQXQOKHGLEKLNWTKSFDQURO3OWB27FQIAPCMMNHC4V3WLADQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.011233614 (323h21m12.473326098s) [noise=true , address=145.79.227.3:4150 , piece_id=OOXPQHWNAKK52AKWDEM7GEH3PPFN2FS5HSBZB7WJZ7VH365DQN6A, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.014548516 (323h21m12.476640999s) [noise=true , address=45.91.116.11:4117 , piece_id=QJIQT6PZ64YCS4MOQCSX5FY3PR4Z2OWACEJJVWXOPB5IV2DJGMVA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.017671924 (323h21m12.479764407s) [noise=true , address=145.79.227.18:4006 , piece_id=5KTVQV4YJ2V3LB7O5V2S5TIFHXBJ5NVJR244MC4KCVK42UPQOGQQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.022176361 (323h21m12.484268834s) [noise=true , address=145.79.227.16:4912 , piece_id=PPVFKSZMR6IU7RIEFAHDUNL453UXSWWCM7G7LP2OGNUEF6QP4HDA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.023896196 (323h21m12.48598867s) [noise=true , address=145.79.227.2:4113 , piece_id=QGNM2PVLQBQZLIJXHMLZMAFEFGVISFRME2DBXHGFXXVRRSFE4IBQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.026829021 (323h21m12.488921494s) [noise=true , address=77.90.31.10:4520 , piece_id=LCZKCLLUDLASEQSFIIWC4QVY3V2GH4BO4LIXKZL6KJEML76QFRXA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.027439743 (323h21m12.489532216s) [noise=true , address=145.79.227.6:4259 , piece_id=D7J5H6YIEN4VIUVYL5QQCIVVRKUH6KCOCUX3OSYV6BIHUPBR3UEQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.029282001 (323h21m12.491374484s) [noise=true , address=45.91.116.8:4048 , piece_id=OKLLUPVCU5VHCIV34L2UJMPUXOB3YZOPR6VJKIQBOF4UAK7P3M2Q, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.030439296 (323h21m12.492531769s) [noise=true , address=77.90.31.13:4628 , piece_id=K23CB3Z57UYPF4UZDQ3JN47LL4PORWUQCFPEPBS5BRFEKADOMB5A, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.032398566 (323h21m12.494491039s) [noise=true , address=145.79.227.14:4840 , piece_id=NP32LN2KO3Q2VT4WGYY53CH6TWHPILFNDJASKUFILTNSULLQNPLQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:28.730821790 (323h21m14.192914273s) [upload=284576]: Next returned nil segment Oct 7 00:52:28.730842921 (323h21m14.192935394s) [upload=284576]: Waiting for error group managing segments... Oct 7 00:52:29.018765363 (323h21m14.480857846s) [noise=true , address=145.79.227.16:4912 , piece_id=PPVFKSZMR6IU7RIEFAHDUNL453UXSWWCM7G7LP2OGNUEF6QP4HDA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.019386345 (323h21m14.481478818s) [noise=true , address=145.79.227.18:4006 , piece_id=5KTVQV4YJ2V3LB7O5V2S5TIFHXBJ5NVJR244MC4KCVK42UPQOGQQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.023241657 (323h21m14.48533414s) [noise=true , address=145.79.227.6:4259 , piece_id=D7J5H6YIEN4VIUVYL5QQCIVVRKUH6KCOCUX3OSYV6BIHUPBR3UEQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.023572917 (323h21m14.48566539s) [noise=true , address=145.79.227.15:4874 , piece_id=5DQFDNCETE743IDGWXSOXWUO3U42RXHRIZ42U2LSBNYFYJ4F73CQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.033328714 (323h21m14.495421187s) [noise=true , address=145.79.227.19:4079 , piece_id=LA6H5KWT45SDLUVXQLYDC2APAYTYIVLNECJEFKZEL3GKIR4RF7CQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.036569455 (323h21m14.498661938s) [noise=true , address=145.79.227.3:4151 , piece_id=CRTQEPDR6WJKBMQZP72CWE65I4ROVJBHFPWAQHPO4NU2IZ6ILNVA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.036935462 (323h21m14.499027945s) [noise=true , address=145.79.227.2:4113 , piece_id=QGNM2PVLQBQZLIJXHMLZMAFEFGVISFRME2DBXHGFXXVRRSFE4IBQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.069794942 (323h21m14.531887415s) [noise=true , address=145.79.227.14:4840 , piece_id=NP32LN2KO3Q2VT4WGYY53CH6TWHPILFNDJASKUFILTNSULLQNPLQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.097526738 (323h21m14.559619211s) [noise=true , address=145.79.227.3:4150 , piece_id=OOXPQHWNAKK52AKWDEM7GEH3PPFN2FS5HSBZB7WJZ7VH365DQN6A, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.151291235 (323h21m14.613383718s) [noise=true , address=45.91.116.5:4298 , piece_id=S6FKHQXQOKHGLEKLNWTKSFDQURO3OWB27FQIAPCMMNHC4V3WLADQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.226239077 (323h21m14.68833155s) [noise=true , address=45.91.116.8:4048 , piece_id=OKLLUPVCU5VHCIV34L2UJMPUXOB3YZOPR6VJKIQBOF4UAK7P3M2Q, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.235900194 (323h21m14.697992667s) [noise=true , address=77.90.31.10:4520 , piece_id=LCZKCLLUDLASEQSFIIWC4QVY3V2GH4BO4LIXKZL6KJEML76QFRXA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.249112243 (323h21m14.711204716s) [noise=true , address=77.90.31.15:4697 , piece_id=S5FQAZPXXKGIZX37YAMAXMIKFYBNBGLHYIK7353NTYOIYXDUIZSA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.249144654 (323h21m14.711237128s) [seg_pos={0 0}, upload=284576]: Segment reached optimal threshold of 13 pieces. Oct 7 00:52:29.249279160 (323h21m14.711371643s) [noise=true , address=45.91.116.11:4117 , piece_id=QJIQT6PZ64YCS4MOQCSX5FY3PR4Z2OWACEJJVWXOPB5IV2DJGMVA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Oct 7 00:52:29.249287907 (323h21m14.71138038s) [noise=true , address=45.91.116.11:4117 , piece_id=QJIQT6PZ64YCS4MOQCSX5FY3PR4Z2OWACEJJVWXOPB5IV2DJGMVA, seg_pos={0 0}, upload=284576]: Long tail cancellation due to optimal threshold Oct 7 00:52:29.249362249 (323h21m14.711454722s) [noise=true , address=45.91.116.7:4009 , piece_id=MPTEURKI6WGHWZVTUZJACJQ646PBNZOM3QUWGBTAYRIS75JCESCQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Oct 7 00:52:29.249367709 (323h21m14.711460182s) [noise=true , address=45.91.116.7:4009 , piece_id=MPTEURKI6WGHWZVTUZJACJQ646PBNZOM3QUWGBTAYRIS75JCESCQ, seg_pos={0 0}, upload=284576]: Long tail cancellation due to optimal threshold Oct 7 00:52:29.249437041 (323h21m14.711529524s) [noise=true , address=77.90.31.13:4628 , piece_id=K23CB3Z57UYPF4UZDQ3JN47LL4PORWUQCFPEPBS5BRFEKADOMB5A, seg_pos={0 0}, upload=284576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Oct 7 00:52:29.249474352 (323h21m14.711566825s) [noise=true , address=77.90.31.13:4628 , piece_id=K23CB3Z57UYPF4UZDQ3JN47LL4PORWUQCFPEPBS5BRFEKADOMB5A, seg_pos={0 0}, upload=284576]: Long tail cancellation due to optimal threshold Oct 7 00:52:29.249504179 (323h21m14.711596652s) [seg_pos={0 0}, upload=284576]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:52:29.249600092 (323h21m14.711692565s) [upload=284576]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:52:29.249606494 (323h21m14.711698967s) [upload=284576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:52:29.613316298 (323h21m15.075408771s) [upload=284576]: Done uploading segments... Oct 7 00:52:29.613340905 (323h21m15.075433378s) [upload=284576]: Upload finished. err: <nil> Duration: 6.167092912s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................X.XXX.] duration: 6.516646129s FAIL (2025-10-07 00:53:15 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.516682539s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 20.245666804s FAIL (2025-10-07 00:53:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 20.245700368s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................X..XX] duration: 30.00199937s FAIL (2025-10-07 00:52:47 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Oct 7 00:52:22.589363902 (323h21m8.051456385s) [upload=284576]: Starting upload Oct 7 00:52:22.589458121 (323h21m8.051550594s) [upload=284576]: Done starting upload Oct 7 00:52:22.589588860 (323h21m8.051681363s) [upload=284576]: Uploading segments... Oct 7 00:52:22.589745138 (323h21m8.051837631s) [upload=284576]: Got next segment. Inline: false Oct 7 00:52:22.589791246 (323h21m8.051883729s) [upload=284576]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:52:22.589818087 (323h21m8.05191057s) [upload=284576]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:52:27.006217443 (323h21m12.468309956s) [seg_pos={0 0}, upload=284576]: Begin upload segment... Oct 7 00:52:27.006331671 (323h21m12.468424154s) [seg_pos={0 0}, upload=284576]: Done begin upload segment. Oct 7 00:52:27.006375454 (323h21m12.468467927s) [noise=true , address=145.79.227.15:4874 , piece_id=5DQFDNCETE743IDGWXSOXWUO3U42RXHRIZ42U2LSBNYFYJ4F73CQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.007934914 (323h21m12.470027387s) [noise=true , address=145.79.227.19:4079 , piece_id=LA6H5KWT45SDLUVXQLYDC2APAYTYIVLNECJEFKZEL3GKIR4RF7CQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.008623895 (323h21m12.470716368s) [noise=true , address=77.90.31.15:4697 , piece_id=S5FQAZPXXKGIZX37YAMAXMIKFYBNBGLHYIK7353NTYOIYXDUIZSA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.008964324 (323h21m12.471056797s) [noise=true , address=145.79.227.3:4151 , piece_id=CRTQEPDR6WJKBMQZP72CWE65I4ROVJBHFPWAQHPO4NU2IZ6ILNVA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.009411014 (323h21m12.471503497s) [noise=true , address=45.91.116.7:4009 , piece_id=MPTEURKI6WGHWZVTUZJACJQ646PBNZOM3QUWGBTAYRIS75JCESCQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.010011507 (323h21m12.472103991s) [noise=true , address=45.91.116.5:4298 , piece_id=S6FKHQXQOKHGLEKLNWTKSFDQURO3OWB27FQIAPCMMNHC4V3WLADQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.011233614 (323h21m12.473326098s) [noise=true , address=145.79.227.3:4150 , piece_id=OOXPQHWNAKK52AKWDEM7GEH3PPFN2FS5HSBZB7WJZ7VH365DQN6A, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.014548516 (323h21m12.476640999s) [noise=true , address=45.91.116.11:4117 , piece_id=QJIQT6PZ64YCS4MOQCSX5FY3PR4Z2OWACEJJVWXOPB5IV2DJGMVA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.017671924 (323h21m12.479764407s) [noise=true , address=145.79.227.18:4006 , piece_id=5KTVQV4YJ2V3LB7O5V2S5TIFHXBJ5NVJR244MC4KCVK42UPQOGQQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.022176361 (323h21m12.484268834s) [noise=true , address=145.79.227.16:4912 , piece_id=PPVFKSZMR6IU7RIEFAHDUNL453UXSWWCM7G7LP2OGNUEF6QP4HDA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.023896196 (323h21m12.48598867s) [noise=true , address=145.79.227.2:4113 , piece_id=QGNM2PVLQBQZLIJXHMLZMAFEFGVISFRME2DBXHGFXXVRRSFE4IBQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.026829021 (323h21m12.488921494s) [noise=true , address=77.90.31.10:4520 , piece_id=LCZKCLLUDLASEQSFIIWC4QVY3V2GH4BO4LIXKZL6KJEML76QFRXA, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.027439743 (323h21m12.489532216s) [noise=true , address=145.79.227.6:4259 , piece_id=D7J5H6YIEN4VIUVYL5QQCIVVRKUH6KCOCUX3OSYV6BIHUPBR3UEQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.029282001 (323h21m12.491374484s) [noise=true , address=45.91.116.8:4048 , piece_id=OKLLUPVCU5VHCIV34L2UJMPUXOB3YZOPR6VJKIQBOF4UAK7P3M2Q, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.030439296 (323h21m12.492531769s) [noise=true , address=77.90.31.13:4628 , piece_id=K23CB3Z57UYPF4UZDQ3JN47LL4PORWUQCFPEPBS5BRFEKADOMB5A, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:27.032398566 (323h21m12.494491039s) [noise=true , address=145.79.227.14:4840 , piece_id=NP32LN2KO3Q2VT4WGYY53CH6TWHPILFNDJASKUFILTNSULLQNPLQ, seg_pos={0 0}, upload=284576]: Uploading piece... Oct 7 00:52:28.730821790 (323h21m14.192914273s) [upload=284576]: Next returned nil segment Oct 7 00:52:28.730842921 (323h21m14.192935394s) [upload=284576]: Waiting for error group managing segments... Oct 7 00:52:29.018765363 (323h21m14.480857846s) [noise=true , address=145.79.227.16:4912 , piece_id=PPVFKSZMR6IU7RIEFAHDUNL453UXSWWCM7G7LP2OGNUEF6QP4HDA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.019386345 (323h21m14.481478818s) [noise=true , address=145.79.227.18:4006 , piece_id=5KTVQV4YJ2V3LB7O5V2S5TIFHXBJ5NVJR244MC4KCVK42UPQOGQQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.023241657 (323h21m14.48533414s) [noise=true , address=145.79.227.6:4259 , piece_id=D7J5H6YIEN4VIUVYL5QQCIVVRKUH6KCOCUX3OSYV6BIHUPBR3UEQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.023572917 (323h21m14.48566539s) [noise=true , address=145.79.227.15:4874 , piece_id=5DQFDNCETE743IDGWXSOXWUO3U42RXHRIZ42U2LSBNYFYJ4F73CQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.033328714 (323h21m14.495421187s) [noise=true , address=145.79.227.19:4079 , piece_id=LA6H5KWT45SDLUVXQLYDC2APAYTYIVLNECJEFKZEL3GKIR4RF7CQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.036569455 (323h21m14.498661938s) [noise=true , address=145.79.227.3:4151 , piece_id=CRTQEPDR6WJKBMQZP72CWE65I4ROVJBHFPWAQHPO4NU2IZ6ILNVA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.036935462 (323h21m14.499027945s) [noise=true , address=145.79.227.2:4113 , piece_id=QGNM2PVLQBQZLIJXHMLZMAFEFGVISFRME2DBXHGFXXVRRSFE4IBQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.069794942 (323h21m14.531887415s) [noise=true , address=145.79.227.14:4840 , piece_id=NP32LN2KO3Q2VT4WGYY53CH6TWHPILFNDJASKUFILTNSULLQNPLQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.097526738 (323h21m14.559619211s) [noise=true , address=145.79.227.3:4150 , piece_id=OOXPQHWNAKK52AKWDEM7GEH3PPFN2FS5HSBZB7WJZ7VH365DQN6A, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.151291235 (323h21m14.613383718s) [noise=true , address=45.91.116.5:4298 , piece_id=S6FKHQXQOKHGLEKLNWTKSFDQURO3OWB27FQIAPCMMNHC4V3WLADQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.226239077 (323h21m14.68833155s) [noise=true , address=45.91.116.8:4048 , piece_id=OKLLUPVCU5VHCIV34L2UJMPUXOB3YZOPR6VJKIQBOF4UAK7P3M2Q, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.235900194 (323h21m14.697992667s) [noise=true , address=77.90.31.10:4520 , piece_id=LCZKCLLUDLASEQSFIIWC4QVY3V2GH4BO4LIXKZL6KJEML76QFRXA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.249112243 (323h21m14.711204716s) [noise=true , address=77.90.31.15:4697 , piece_id=S5FQAZPXXKGIZX37YAMAXMIKFYBNBGLHYIK7353NTYOIYXDUIZSA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: <nil> Oct 7 00:52:29.249144654 (323h21m14.711237128s) [seg_pos={0 0}, upload=284576]: Segment reached optimal threshold of 13 pieces. Oct 7 00:52:29.249279160 (323h21m14.711371643s) [noise=true , address=45.91.116.11:4117 , piece_id=QJIQT6PZ64YCS4MOQCSX5FY3PR4Z2OWACEJJVWXOPB5IV2DJGMVA, seg_pos={0 0}, upload=284576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Oct 7 00:52:29.249287907 (323h21m14.71138038s) [noise=true , address=45.91.116.11:4117 , piece_id=QJIQT6PZ64YCS4MOQCSX5FY3PR4Z2OWACEJJVWXOPB5IV2DJGMVA, seg_pos={0 0}, upload=284576]: Long tail cancellation due to optimal threshold Oct 7 00:52:29.249362249 (323h21m14.711454722s) [noise=true , address=45.91.116.7:4009 , piece_id=MPTEURKI6WGHWZVTUZJACJQ646PBNZOM3QUWGBTAYRIS75JCESCQ, seg_pos={0 0}, upload=284576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Oct 7 00:52:29.249367709 (323h21m14.711460182s) [noise=true , address=45.91.116.7:4009 , piece_id=MPTEURKI6WGHWZVTUZJACJQ646PBNZOM3QUWGBTAYRIS75JCESCQ, seg_pos={0 0}, upload=284576]: Long tail cancellation due to optimal threshold Oct 7 00:52:29.249437041 (323h21m14.711529524s) [noise=true , address=77.90.31.13:4628 , piece_id=K23CB3Z57UYPF4UZDQ3JN47LL4PORWUQCFPEPBS5BRFEKADOMB5A, seg_pos={0 0}, upload=284576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a): protocol: expected piece hash; context canceled; EOF Oct 7 00:52:29.249474352 (323h21m14.711566825s) [noise=true , address=77.90.31.13:4628 , piece_id=K23CB3Z57UYPF4UZDQ3JN47LL4PORWUQCFPEPBS5BRFEKADOMB5A, seg_pos={0 0}, upload=284576]: Long tail cancellation due to optimal threshold Oct 7 00:52:29.249504179 (323h21m14.711596652s) [seg_pos={0 0}, upload=284576]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:52:29.249600092 (323h21m14.711692565s) [upload=284576]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:52:29.249606494 (323h21m14.711698967s) [upload=284576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:52:29.613316298 (323h21m15.075408771s) [upload=284576]: Done uploading segments... Oct 7 00:52:29.613340905 (323h21m15.075433378s) [upload=284576]: Upload finished. err: <nil> Duration: 30.002032493s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................X.XXX] duration: 30.000744558s FAIL (2025-10-07 00:52:39 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000794913s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................X.XX] duration: 30.000977444s FAIL (2025-10-07 00:52:09 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001005637s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 18.71892908s FAIL (2025-10-07 00:52:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 18.718961071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X..X] duration: 30.002552225s FAIL (2025-10-07 00:51:36 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Oct 7 00:51:07.079454302 (323h19m52.541546785s) [upload=284560]: Starting upload Oct 7 00:51:07.079557789 (323h19m52.541650272s) [upload=284560]: Done starting upload Oct 7 00:51:07.079637089 (323h19m52.541729572s) [upload=284560]: Uploading segments... Oct 7 00:51:07.080011853 (323h19m52.542104336s) [upload=284560]: Got next segment. Inline: false Oct 7 00:51:07.080057139 (323h19m52.542149622s) [upload=284560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:51:07.080063321 (323h19m52.542155804s) [upload=284560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:51:33.025887653 (323h20m18.487980156s) [seg_pos={0 0}, upload=284560]: Begin upload segment... Oct 7 00:51:33.025963458 (323h20m18.488055931s) [seg_pos={0 0}, upload=284560]: Done begin upload segment. Oct 7 00:51:33.025996160 (323h20m18.488088634s) [noise=true , address=45.91.116.11:4118 , piece_id=EUTAWK3JE6ZCZMLGEJKYDEBSN7Q3WKATOEO5BD4CP37QESDFESQQ, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.026089828 (323h20m18.488182331s) [noise=true , address=45.91.116.8:4047 , piece_id=USZEPY64NMNWKP3GNGNUQYKHRYRY6LVU75ND7VMFFU76PPRZNN5Q, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.026665805 (323h20m18.488758278s) [noise=true , address=145.79.227.13:4804 , piece_id=6XJIFPH65CZUDEUX64MHMQBFULYPXUVCBO52WYLIOKQ66W5K2XUQ, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.027141712 (323h20m18.489234195s) [noise=true , address=45.91.116.4:4940 , piece_id=VSXLWPNWVAQIBM65MFQE6DJ6CA43QQ3Q6I6GBZVCNLCOJ32NHTRA, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.027500775 (323h20m18.489593248s) [noise=true , address=77.90.31.18:4805 , piece_id=OYBKV3TXARGRMUQMI72DER4UNMZKXCMGFH36K6F2OANQC3V7NY4Q, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.027817879 (323h20m18.489910352s) [noise=true , address=145.79.227.2:4113 , piece_id=VAUESXPQVU2IQYDZULKXA43KC6ZKHNYLSBH6XKDNPT2ZBWYLHACA, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.027864988 (323h20m18.489957471s) [noise=true , address=145.79.227.11:4729 , piece_id=TJYZW5Q4Q34LKTO4KNT3BIEKGOOH2633Y7QXQMQBGCV546TDMNPQ, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.028078495 (323h20m18.490170968s) [noise=true , address=145.79.227.5:4223 , piece_id=72KWD2RIOJMBEDSWFXVMF53QVLOSPLMMTYNJ7K65DZOZTFVN3CPQ, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.028359019 (323h20m18.490451492s) [noise=true , address=77.90.31.16:4735 , piece_id=2NELHIFYICWC7JCDXB5M2MAELZ34WVQZ3PWGVI77ZWY3QA4FLTDQ, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.028393064 (323h20m18.490485537s) [noise=true , address=77.90.31.11:4554 , piece_id=TUC6E47W7ZQKBRGS67HFB6GQECOL3477QF3SBKKEXSPQRGODS65A, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.028969993 (323h20m18.491062476s) [noise=true , address=77.90.31.21:4879 , piece_id=FJEQZBYBYMV5UGJNIQQQMTIRGS52FRFOJDIR5GRCSF6J6Q45YJSQ, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.029400562 (323h20m18.491493045s) [noise=true , address=45.91.116.12:4226 , piece_id=T4QZPSISIRPNIHZEU3VJKDNZHEBUWATHGYWLJRERAOKDDZEYE4ZA, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.029627805 (323h20m18.491720278s) [noise=true , address=77.90.31.10:4517 , piece_id=Z5J3UEY33POHXKTXAF26Y3VZZFM46ZLPDD76DYX6CFCYLLA74GDA, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.029994223 (323h20m18.492086706s) [noise=true , address=77.90.31.13:4626 , piece_id=VTYR7PC6K63N74BABLN547DFUSKTG6JDCI3TDDMZ3G443Z3T6W5A, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.030303852 (323h20m18.492396325s) [noise=true , address=45.91.116.14:4189 , piece_id=XMVJY3Z3A5GZLGI7LMTAKHS4V3DUS7P5FTJ4EPN3ALMJ4UUX64PA, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.030591411 (323h20m18.492683884s) [noise=true , address=145.79.227.7:4294 , piece_id=TWWMBRICPDQOBG2GWNEK5EI6EHLZ6SRGNP7NJE7MRR3PCD7UXLLQ, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:34.726921950 (323h20m20.189014423s) [upload=284560]: Next returned nil segment Oct 7 00:51:34.726942679 (323h20m20.189035162s) [upload=284560]: Waiting for error group managing segments... Oct 7 00:51:35.017547750 (323h20m20.479640223s) [noise=true , address=145.79.227.2:4113 , piece_id=VAUESXPQVU2IQYDZULKXA43KC6ZKHNYLSBH6XKDNPT2ZBWYLHACA, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.020695543 (323h20m20.482788026s) [noise=true , address=145.79.227.13:4804 , piece_id=6XJIFPH65CZUDEUX64MHMQBFULYPXUVCBO52WYLIOKQ66W5K2XUQ, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.021033347 (323h20m20.48312584s) [noise=true , address=145.79.227.11:4729 , piece_id=TJYZW5Q4Q34LKTO4KNT3BIEKGOOH2633Y7QXQMQBGCV546TDMNPQ, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.031917472 (323h20m20.494009945s) [noise=true , address=145.79.227.7:4294 , piece_id=TWWMBRICPDQOBG2GWNEK5EI6EHLZ6SRGNP7NJE7MRR3PCD7UXLLQ, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.038490138 (323h20m20.500582621s) [noise=true , address=145.79.227.5:4223 , piece_id=72KWD2RIOJMBEDSWFXVMF53QVLOSPLMMTYNJ7K65DZOZTFVN3CPQ, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.126945902 (323h20m20.589038375s) [noise=true , address=45.91.116.4:4940 , piece_id=VSXLWPNWVAQIBM65MFQE6DJ6CA43QQ3Q6I6GBZVCNLCOJ32NHTRA, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.143291566 (323h20m20.605384039s) [noise=true , address=45.91.116.12:4226 , piece_id=T4QZPSISIRPNIHZEU3VJKDNZHEBUWATHGYWLJRERAOKDDZEYE4ZA, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.147072747 (323h20m20.60916523s) [noise=true , address=77.90.31.10:4517 , piece_id=Z5J3UEY33POHXKTXAF26Y3VZZFM46ZLPDD76DYX6CFCYLLA74GDA, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.161622793 (323h20m20.623715266s) [noise=true , address=77.90.31.16:4735 , piece_id=2NELHIFYICWC7JCDXB5M2MAELZ34WVQZ3PWGVI77ZWY3QA4FLTDQ, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.163300327 (323h20m20.62539281s) [noise=true , address=77.90.31.18:4805 , piece_id=OYBKV3TXARGRMUQMI72DER4UNMZKXCMGFH36K6F2OANQC3V7NY4Q, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.167154646 (323h20m20.629247129s) [noise=true , address=77.90.31.21:4879 , piece_id=FJEQZBYBYMV5UGJNIQQQMTIRGS52FRFOJDIR5GRCSF6J6Q45YJSQ, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.217288910 (323h20m20.679381394s) [noise=true , address=77.90.31.11:4554 , piece_id=TUC6E47W7ZQKBRGS67HFB6GQECOL3477QF3SBKKEXSPQRGODS65A, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.219923928 (323h20m20.682016411s) [noise=true , address=77.90.31.13:4626 , piece_id=VTYR7PC6K63N74BABLN547DFUSKTG6JDCI3TDDMZ3G443Z3T6W5A, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.219934047 (323h20m20.68202653s) [seg_pos={0 0}, upload=284560]: Segment reached optimal threshold of 13 pieces. Oct 7 00:51:35.220097188 (323h20m20.682189661s) [noise=true , address=45.91.116.8:4047 , piece_id=USZEPY64NMNWKP3GNGNUQYKHRYRY6LVU75ND7VMFFU76PPRZNN5Q, seg_pos={0 0}, upload=284560]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Oct 7 00:51:35.220106335 (323h20m20.682198818s) [noise=true , address=45.91.116.8:4047 , piece_id=USZEPY64NMNWKP3GNGNUQYKHRYRY6LVU75ND7VMFFU76PPRZNN5Q, seg_pos={0 0}, upload=284560]: Long tail cancellation due to optimal threshold Oct 7 00:51:35.220185816 (323h20m20.6822783s) [noise=true , address=45.91.116.14:4189 , piece_id=XMVJY3Z3A5GZLGI7LMTAKHS4V3DUS7P5FTJ4EPN3ALMJ4UUX64PA, seg_pos={0 0}, upload=284560]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Oct 7 00:51:35.220191547 (323h20m20.68228403s) [noise=true , address=45.91.116.14:4189 , piece_id=XMVJY3Z3A5GZLGI7LMTAKHS4V3DUS7P5FTJ4EPN3ALMJ4UUX64PA, seg_pos={0 0}, upload=284560]: Long tail cancellation due to optimal threshold Oct 7 00:51:35.220246823 (323h20m20.682339306s) [noise=true , address=45.91.116.11:4118 , piece_id=EUTAWK3JE6ZCZMLGEJKYDEBSN7Q3WKATOEO5BD4CP37QESDFESQQ, seg_pos={0 0}, upload=284560]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Oct 7 00:51:35.220253205 (323h20m20.682345688s) [noise=true , address=45.91.116.11:4118 , piece_id=EUTAWK3JE6ZCZMLGEJKYDEBSN7Q3WKATOEO5BD4CP37QESDFESQQ, seg_pos={0 0}, upload=284560]: Long tail cancellation due to optimal threshold Oct 7 00:51:35.220272602 (323h20m20.682365075s) [seg_pos={0 0}, upload=284560]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:51:35.220355991 (323h20m20.682448464s) [upload=284560]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:51:35.220397750 (323h20m20.682490233s) [upload=284560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:51:35.601020299 (323h20m21.063112782s) [upload=284560]: Done uploading segments... Oct 7 00:51:35.601045828 (323h20m21.063138311s) [upload=284560]: Upload finished. err: <nil> Duration: 30.002579867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 30.001610917s FAIL (2025-10-07 00:51:04 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.00162794s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 25.787861236s FAIL (2025-10-07 00:51:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 25.787916872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 9.783680143s FAIL (2025-10-07 00:50:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 00:48:56.613229313 (323h17m42.075321796s) [upload=284523]: Starting upload Oct 7 00:48:56.613324523 (323h17m42.075417007s) [upload=284523]: Done starting upload Oct 7 00:48:56.613384268 (323h17m42.075476751s) [upload=284523]: Uploading segments... Oct 7 00:48:56.613615168 (323h17m42.075707641s) [upload=284523]: Got next segment. Inline: false Oct 7 00:48:56.613663609 (323h17m42.075756082s) [upload=284523]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:48:56.613669240 (323h17m42.075761723s) [upload=284523]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:48:57.190575036 (323h17m42.652667519s) [seg_pos={0 0}, upload=284523]: Begin upload segment... Oct 7 00:48:57.190638877 (323h17m42.65273136s) [seg_pos={0 0}, upload=284523]: Done begin upload segment. Oct 7 00:48:57.190666610 (323h17m42.652759083s) [noise=true , address=145.79.227.19:4079 , piece_id=N2HVTF7Z5LWHTUU2XP5U7H6OGZE5FPLGXWBM6LUBV4S63OYCKIZQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.190709111 (323h17m42.652801594s) [noise=true , address=145.79.227.16:4909 , piece_id=AF5U7EEX66OXIUGORRI7PARX3GIUHMYWEWV4IQRQG2UBHJVUKNZA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191214002 (323h17m42.653306475s) [noise=true , address=145.79.227.12:4767 , piece_id=FAUAYZLZS3NHJSVOG4G6OSWCIYFMA2V4TYBAKVHXEIEKKTCCVYEA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191500899 (323h17m42.653593372s) [noise=true , address=45.91.116.8:4048 , piece_id=WAWXPIYZT5FJDN3WMDLBYXXVFYG2RQQC6ZY3UJWHVGFOQTOVXPLQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191879570 (323h17m42.653972063s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.192213936 (323h17m42.654306409s) [noise=true , address=145.79.227.14:4837 , piece_id=F4TSTSTHX5BX3OADIIGUIFS26TXCFQPGVS75VLB3HQEABEX6AT7A, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.192648384 (323h17m42.654740867s) [noise=true , address=145.79.227.3:4151 , piece_id=POHBULLPCNQDQYDXHQO5IERYITYGXCYQQMMKJDAN6ZB7BJILYCZQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.193009221 (323h17m42.655101694s) [noise=true , address=145.79.227.18:4005 , piece_id=7Y2NPR5TOSPDZHGK5GWKI7WRW55W7YFQFOWS3Z4A4LSW7RUYU7LA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.193453617 (323h17m42.6555461s) [noise=true , address=145.79.227.4:4186 , piece_id=O6JFC6Q75TSMLYEYMN3LGCSDSCH7IZQZGGR4ZDHN7PNLPXWR46TQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.194729406 (323h17m42.656821889s) [noise=true , address=77.90.31.21:4877 , piece_id=KUWJQN5THELOZ2DCE7WVN23UDCTOZEMOPLRQF36QWKHKZEQBQYKA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.196132077 (323h17m42.658224561s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.198421616 (323h17m42.660514099s) [noise=true , address=45.91.116.13:4155 , piece_id=LQLGG276QYLAFCV7N25H4I25JXR4QXP647OUFIOZG4DZ7EVNFFFA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.200790979 (323h17m42.662883452s) [noise=true , address=145.79.227.2:4113 , piece_id=R3XEJLWII4XLIESE46RJBOYQESBJKC6P7SHXQ64O4UQZITPZMFJQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.202469044 (323h17m42.664561527s) [noise=true , address=145.79.227.16:4911 , piece_id=P6Q6O6VBBY5F4S774JGMH3BXQYW46PRQWWXNOKM4YCTXOLADOBLA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.202952645 (323h17m42.665045128s) [noise=true , address=145.79.227.17:4972 , piece_id=KXVAP72FVEAHTZ6RQH5I5Q3LYKRBUUUZZOKZWOLLCBZHAYCZKTMQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.204311693 (323h17m42.666404176s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:58.852892147 (323h17m44.31498463s) [upload=284523]: Next returned nil segment Oct 7 00:48:58.852911884 (323h17m44.315004367s) [upload=284523]: Waiting for error group managing segments... Oct 7 00:48:59.177446373 (323h17m44.639538846s) [noise=true , address=145.79.227.12:4767 , piece_id=FAUAYZLZS3NHJSVOG4G6OSWCIYFMA2V4TYBAKVHXEIEKKTCCVYEA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.180797364 (323h17m44.642889847s) [noise=true , address=145.79.227.19:4079 , piece_id=N2HVTF7Z5LWHTUU2XP5U7H6OGZE5FPLGXWBM6LUBV4S63OYCKIZQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.185354562 (323h17m44.647447045s) [noise=true , address=145.79.227.18:4005 , piece_id=7Y2NPR5TOSPDZHGK5GWKI7WRW55W7YFQFOWS3Z4A4LSW7RUYU7LA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.185828054 (323h17m44.647920527s) [noise=true , address=145.79.227.4:4186 , piece_id=O6JFC6Q75TSMLYEYMN3LGCSDSCH7IZQZGGR4ZDHN7PNLPXWR46TQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.190734186 (323h17m44.652826659s) [noise=true , address=145.79.227.17:4972 , piece_id=KXVAP72FVEAHTZ6RQH5I5Q3LYKRBUUUZZOKZWOLLCBZHAYCZKTMQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.192115355 (323h17m44.654207838s) [noise=true , address=145.79.227.16:4909 , piece_id=AF5U7EEX66OXIUGORRI7PARX3GIUHMYWEWV4IQRQG2UBHJVUKNZA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.201433238 (323h17m44.663525711s) [noise=true , address=145.79.227.3:4151 , piece_id=POHBULLPCNQDQYDXHQO5IERYITYGXCYQQMMKJDAN6ZB7BJILYCZQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.202680584 (323h17m44.664773057s) [noise=true , address=145.79.227.14:4837 , piece_id=F4TSTSTHX5BX3OADIIGUIFS26TXCFQPGVS75VLB3HQEABEX6AT7A, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.208003650 (323h17m44.670096123s) [noise=true , address=145.79.227.2:4113 , piece_id=R3XEJLWII4XLIESE46RJBOYQESBJKC6P7SHXQ64O4UQZITPZMFJQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.245499177 (323h17m44.70759165s) [noise=true , address=145.79.227.16:4911 , piece_id=P6Q6O6VBBY5F4S774JGMH3BXQYW46PRQWWXNOKM4YCTXOLADOBLA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.337714280 (323h17m44.799806763s) [noise=true , address=45.91.116.13:4155 , piece_id=LQLGG276QYLAFCV7N25H4I25JXR4QXP647OUFIOZG4DZ7EVNFFFA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.359104120 (323h17m44.821196604s) [noise=true , address=45.91.116.8:4048 , piece_id=WAWXPIYZT5FJDN3WMDLBYXXVFYG2RQQC6ZY3UJWHVGFOQTOVXPLQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.360706612 (323h17m44.822799095s) [noise=true , address=77.90.31.21:4877 , piece_id=KUWJQN5THELOZ2DCE7WVN23UDCTOZEMOPLRQF36QWKHKZEQBQYKA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.360777166 (323h17m44.822869659s) [seg_pos={0 0}, upload=284523]: Segment reached optimal threshold of 13 pieces. Oct 7 00:48:59.360932752 (323h17m44.823025225s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.360944375 (323h17m44.823036848s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361053563 (323h17m44.823146036s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.361062389 (323h17m44.823154872s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361107885 (323h17m44.823200359s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.361129166 (323h17m44.823221649s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361162901 (323h17m44.823255374s) [seg_pos={0 0}, upload=284523]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:48:59.361244957 (323h17m44.82333743s) [upload=284523]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:48:59.361250958 (323h17m44.823343431s) [upload=284523]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:48:59.734954893 (323h17m45.197047386s) [upload=284523]: Done uploading segments... Oct 7 00:48:59.734978559 (323h17m45.197071052s) [upload=284523]: Upload finished. err: <nil> Duration: 9.783711585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 19.121502596s FAIL (2025-10-07 00:50:33 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 19.121535799s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 22.75882405s FAIL (2025-10-07 00:50:15 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 22.758848075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.002337474s FAIL (2025-10-07 00:49:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 1m0.002373843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 25.262001788s FAIL (2025-10-07 00:49:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 00:48:56.613229313 (323h17m42.075321796s) [upload=284523]: Starting upload Oct 7 00:48:56.613324523 (323h17m42.075417007s) [upload=284523]: Done starting upload Oct 7 00:48:56.613384268 (323h17m42.075476751s) [upload=284523]: Uploading segments... Oct 7 00:48:56.613615168 (323h17m42.075707641s) [upload=284523]: Got next segment. Inline: false Oct 7 00:48:56.613663609 (323h17m42.075756082s) [upload=284523]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:48:56.613669240 (323h17m42.075761723s) [upload=284523]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:48:57.190575036 (323h17m42.652667519s) [seg_pos={0 0}, upload=284523]: Begin upload segment... Oct 7 00:48:57.190638877 (323h17m42.65273136s) [seg_pos={0 0}, upload=284523]: Done begin upload segment. Oct 7 00:48:57.190666610 (323h17m42.652759083s) [noise=true , address=145.79.227.19:4079 , piece_id=N2HVTF7Z5LWHTUU2XP5U7H6OGZE5FPLGXWBM6LUBV4S63OYCKIZQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.190709111 (323h17m42.652801594s) [noise=true , address=145.79.227.16:4909 , piece_id=AF5U7EEX66OXIUGORRI7PARX3GIUHMYWEWV4IQRQG2UBHJVUKNZA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191214002 (323h17m42.653306475s) [noise=true , address=145.79.227.12:4767 , piece_id=FAUAYZLZS3NHJSVOG4G6OSWCIYFMA2V4TYBAKVHXEIEKKTCCVYEA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191500899 (323h17m42.653593372s) [noise=true , address=45.91.116.8:4048 , piece_id=WAWXPIYZT5FJDN3WMDLBYXXVFYG2RQQC6ZY3UJWHVGFOQTOVXPLQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191879570 (323h17m42.653972063s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.192213936 (323h17m42.654306409s) [noise=true , address=145.79.227.14:4837 , piece_id=F4TSTSTHX5BX3OADIIGUIFS26TXCFQPGVS75VLB3HQEABEX6AT7A, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.192648384 (323h17m42.654740867s) [noise=true , address=145.79.227.3:4151 , piece_id=POHBULLPCNQDQYDXHQO5IERYITYGXCYQQMMKJDAN6ZB7BJILYCZQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.193009221 (323h17m42.655101694s) [noise=true , address=145.79.227.18:4005 , piece_id=7Y2NPR5TOSPDZHGK5GWKI7WRW55W7YFQFOWS3Z4A4LSW7RUYU7LA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.193453617 (323h17m42.6555461s) [noise=true , address=145.79.227.4:4186 , piece_id=O6JFC6Q75TSMLYEYMN3LGCSDSCH7IZQZGGR4ZDHN7PNLPXWR46TQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.194729406 (323h17m42.656821889s) [noise=true , address=77.90.31.21:4877 , piece_id=KUWJQN5THELOZ2DCE7WVN23UDCTOZEMOPLRQF36QWKHKZEQBQYKA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.196132077 (323h17m42.658224561s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.198421616 (323h17m42.660514099s) [noise=true , address=45.91.116.13:4155 , piece_id=LQLGG276QYLAFCV7N25H4I25JXR4QXP647OUFIOZG4DZ7EVNFFFA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.200790979 (323h17m42.662883452s) [noise=true , address=145.79.227.2:4113 , piece_id=R3XEJLWII4XLIESE46RJBOYQESBJKC6P7SHXQ64O4UQZITPZMFJQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.202469044 (323h17m42.664561527s) [noise=true , address=145.79.227.16:4911 , piece_id=P6Q6O6VBBY5F4S774JGMH3BXQYW46PRQWWXNOKM4YCTXOLADOBLA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.202952645 (323h17m42.665045128s) [noise=true , address=145.79.227.17:4972 , piece_id=KXVAP72FVEAHTZ6RQH5I5Q3LYKRBUUUZZOKZWOLLCBZHAYCZKTMQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.204311693 (323h17m42.666404176s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:58.852892147 (323h17m44.31498463s) [upload=284523]: Next returned nil segment Oct 7 00:48:58.852911884 (323h17m44.315004367s) [upload=284523]: Waiting for error group managing segments... Oct 7 00:48:59.177446373 (323h17m44.639538846s) [noise=true , address=145.79.227.12:4767 , piece_id=FAUAYZLZS3NHJSVOG4G6OSWCIYFMA2V4TYBAKVHXEIEKKTCCVYEA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.180797364 (323h17m44.642889847s) [noise=true , address=145.79.227.19:4079 , piece_id=N2HVTF7Z5LWHTUU2XP5U7H6OGZE5FPLGXWBM6LUBV4S63OYCKIZQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.185354562 (323h17m44.647447045s) [noise=true , address=145.79.227.18:4005 , piece_id=7Y2NPR5TOSPDZHGK5GWKI7WRW55W7YFQFOWS3Z4A4LSW7RUYU7LA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.185828054 (323h17m44.647920527s) [noise=true , address=145.79.227.4:4186 , piece_id=O6JFC6Q75TSMLYEYMN3LGCSDSCH7IZQZGGR4ZDHN7PNLPXWR46TQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.190734186 (323h17m44.652826659s) [noise=true , address=145.79.227.17:4972 , piece_id=KXVAP72FVEAHTZ6RQH5I5Q3LYKRBUUUZZOKZWOLLCBZHAYCZKTMQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.192115355 (323h17m44.654207838s) [noise=true , address=145.79.227.16:4909 , piece_id=AF5U7EEX66OXIUGORRI7PARX3GIUHMYWEWV4IQRQG2UBHJVUKNZA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.201433238 (323h17m44.663525711s) [noise=true , address=145.79.227.3:4151 , piece_id=POHBULLPCNQDQYDXHQO5IERYITYGXCYQQMMKJDAN6ZB7BJILYCZQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.202680584 (323h17m44.664773057s) [noise=true , address=145.79.227.14:4837 , piece_id=F4TSTSTHX5BX3OADIIGUIFS26TXCFQPGVS75VLB3HQEABEX6AT7A, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.208003650 (323h17m44.670096123s) [noise=true , address=145.79.227.2:4113 , piece_id=R3XEJLWII4XLIESE46RJBOYQESBJKC6P7SHXQ64O4UQZITPZMFJQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.245499177 (323h17m44.70759165s) [noise=true , address=145.79.227.16:4911 , piece_id=P6Q6O6VBBY5F4S774JGMH3BXQYW46PRQWWXNOKM4YCTXOLADOBLA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.337714280 (323h17m44.799806763s) [noise=true , address=45.91.116.13:4155 , piece_id=LQLGG276QYLAFCV7N25H4I25JXR4QXP647OUFIOZG4DZ7EVNFFFA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.359104120 (323h17m44.821196604s) [noise=true , address=45.91.116.8:4048 , piece_id=WAWXPIYZT5FJDN3WMDLBYXXVFYG2RQQC6ZY3UJWHVGFOQTOVXPLQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.360706612 (323h17m44.822799095s) [noise=true , address=77.90.31.21:4877 , piece_id=KUWJQN5THELOZ2DCE7WVN23UDCTOZEMOPLRQF36QWKHKZEQBQYKA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.360777166 (323h17m44.822869659s) [seg_pos={0 0}, upload=284523]: Segment reached optimal threshold of 13 pieces. Oct 7 00:48:59.360932752 (323h17m44.823025225s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.360944375 (323h17m44.823036848s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361053563 (323h17m44.823146036s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.361062389 (323h17m44.823154872s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361107885 (323h17m44.823200359s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.361129166 (323h17m44.823221649s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361162901 (323h17m44.823255374s) [seg_pos={0 0}, upload=284523]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:48:59.361244957 (323h17m44.82333743s) [upload=284523]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:48:59.361250958 (323h17m44.823343431s) [upload=284523]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:48:59.734954893 (323h17m45.197047386s) [upload=284523]: Done uploading segments... Oct 7 00:48:59.734978559 (323h17m45.197071052s) [upload=284523]: Upload finished. err: <nil> Duration: 25.262021866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.004333381s FAIL (2025-10-07 00:49:43 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.004355063s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)