Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 18.71892908s FAIL (2025-10-07 00:52:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 18.718961071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X..X] duration: 30.002552225s FAIL (2025-10-07 00:51:36 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Oct 7 00:51:07.079454302 (323h19m52.541546785s) [upload=284560]: Starting upload Oct 7 00:51:07.079557789 (323h19m52.541650272s) [upload=284560]: Done starting upload Oct 7 00:51:07.079637089 (323h19m52.541729572s) [upload=284560]: Uploading segments... Oct 7 00:51:07.080011853 (323h19m52.542104336s) [upload=284560]: Got next segment. Inline: false Oct 7 00:51:07.080057139 (323h19m52.542149622s) [upload=284560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:51:07.080063321 (323h19m52.542155804s) [upload=284560]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:51:33.025887653 (323h20m18.487980156s) [seg_pos={0 0}, upload=284560]: Begin upload segment... Oct 7 00:51:33.025963458 (323h20m18.488055931s) [seg_pos={0 0}, upload=284560]: Done begin upload segment. Oct 7 00:51:33.025996160 (323h20m18.488088634s) [noise=true , address=45.91.116.11:4118 , piece_id=EUTAWK3JE6ZCZMLGEJKYDEBSN7Q3WKATOEO5BD4CP37QESDFESQQ, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.026089828 (323h20m18.488182331s) [noise=true , address=45.91.116.8:4047 , piece_id=USZEPY64NMNWKP3GNGNUQYKHRYRY6LVU75ND7VMFFU76PPRZNN5Q, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.026665805 (323h20m18.488758278s) [noise=true , address=145.79.227.13:4804 , piece_id=6XJIFPH65CZUDEUX64MHMQBFULYPXUVCBO52WYLIOKQ66W5K2XUQ, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.027141712 (323h20m18.489234195s) [noise=true , address=45.91.116.4:4940 , piece_id=VSXLWPNWVAQIBM65MFQE6DJ6CA43QQ3Q6I6GBZVCNLCOJ32NHTRA, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.027500775 (323h20m18.489593248s) [noise=true , address=77.90.31.18:4805 , piece_id=OYBKV3TXARGRMUQMI72DER4UNMZKXCMGFH36K6F2OANQC3V7NY4Q, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.027817879 (323h20m18.489910352s) [noise=true , address=145.79.227.2:4113 , piece_id=VAUESXPQVU2IQYDZULKXA43KC6ZKHNYLSBH6XKDNPT2ZBWYLHACA, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.027864988 (323h20m18.489957471s) [noise=true , address=145.79.227.11:4729 , piece_id=TJYZW5Q4Q34LKTO4KNT3BIEKGOOH2633Y7QXQMQBGCV546TDMNPQ, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.028078495 (323h20m18.490170968s) [noise=true , address=145.79.227.5:4223 , piece_id=72KWD2RIOJMBEDSWFXVMF53QVLOSPLMMTYNJ7K65DZOZTFVN3CPQ, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.028359019 (323h20m18.490451492s) [noise=true , address=77.90.31.16:4735 , piece_id=2NELHIFYICWC7JCDXB5M2MAELZ34WVQZ3PWGVI77ZWY3QA4FLTDQ, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.028393064 (323h20m18.490485537s) [noise=true , address=77.90.31.11:4554 , piece_id=TUC6E47W7ZQKBRGS67HFB6GQECOL3477QF3SBKKEXSPQRGODS65A, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.028969993 (323h20m18.491062476s) [noise=true , address=77.90.31.21:4879 , piece_id=FJEQZBYBYMV5UGJNIQQQMTIRGS52FRFOJDIR5GRCSF6J6Q45YJSQ, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.029400562 (323h20m18.491493045s) [noise=true , address=45.91.116.12:4226 , piece_id=T4QZPSISIRPNIHZEU3VJKDNZHEBUWATHGYWLJRERAOKDDZEYE4ZA, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.029627805 (323h20m18.491720278s) [noise=true , address=77.90.31.10:4517 , piece_id=Z5J3UEY33POHXKTXAF26Y3VZZFM46ZLPDD76DYX6CFCYLLA74GDA, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.029994223 (323h20m18.492086706s) [noise=true , address=77.90.31.13:4626 , piece_id=VTYR7PC6K63N74BABLN547DFUSKTG6JDCI3TDDMZ3G443Z3T6W5A, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.030303852 (323h20m18.492396325s) [noise=true , address=45.91.116.14:4189 , piece_id=XMVJY3Z3A5GZLGI7LMTAKHS4V3DUS7P5FTJ4EPN3ALMJ4UUX64PA, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:33.030591411 (323h20m18.492683884s) [noise=true , address=145.79.227.7:4294 , piece_id=TWWMBRICPDQOBG2GWNEK5EI6EHLZ6SRGNP7NJE7MRR3PCD7UXLLQ, seg_pos={0 0}, upload=284560]: Uploading piece... Oct 7 00:51:34.726921950 (323h20m20.189014423s) [upload=284560]: Next returned nil segment Oct 7 00:51:34.726942679 (323h20m20.189035162s) [upload=284560]: Waiting for error group managing segments... Oct 7 00:51:35.017547750 (323h20m20.479640223s) [noise=true , address=145.79.227.2:4113 , piece_id=VAUESXPQVU2IQYDZULKXA43KC6ZKHNYLSBH6XKDNPT2ZBWYLHACA, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.020695543 (323h20m20.482788026s) [noise=true , address=145.79.227.13:4804 , piece_id=6XJIFPH65CZUDEUX64MHMQBFULYPXUVCBO52WYLIOKQ66W5K2XUQ, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.021033347 (323h20m20.48312584s) [noise=true , address=145.79.227.11:4729 , piece_id=TJYZW5Q4Q34LKTO4KNT3BIEKGOOH2633Y7QXQMQBGCV546TDMNPQ, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.031917472 (323h20m20.494009945s) [noise=true , address=145.79.227.7:4294 , piece_id=TWWMBRICPDQOBG2GWNEK5EI6EHLZ6SRGNP7NJE7MRR3PCD7UXLLQ, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.038490138 (323h20m20.500582621s) [noise=true , address=145.79.227.5:4223 , piece_id=72KWD2RIOJMBEDSWFXVMF53QVLOSPLMMTYNJ7K65DZOZTFVN3CPQ, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.126945902 (323h20m20.589038375s) [noise=true , address=45.91.116.4:4940 , piece_id=VSXLWPNWVAQIBM65MFQE6DJ6CA43QQ3Q6I6GBZVCNLCOJ32NHTRA, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.143291566 (323h20m20.605384039s) [noise=true , address=45.91.116.12:4226 , piece_id=T4QZPSISIRPNIHZEU3VJKDNZHEBUWATHGYWLJRERAOKDDZEYE4ZA, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.147072747 (323h20m20.60916523s) [noise=true , address=77.90.31.10:4517 , piece_id=Z5J3UEY33POHXKTXAF26Y3VZZFM46ZLPDD76DYX6CFCYLLA74GDA, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.161622793 (323h20m20.623715266s) [noise=true , address=77.90.31.16:4735 , piece_id=2NELHIFYICWC7JCDXB5M2MAELZ34WVQZ3PWGVI77ZWY3QA4FLTDQ, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.163300327 (323h20m20.62539281s) [noise=true , address=77.90.31.18:4805 , piece_id=OYBKV3TXARGRMUQMI72DER4UNMZKXCMGFH36K6F2OANQC3V7NY4Q, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.167154646 (323h20m20.629247129s) [noise=true , address=77.90.31.21:4879 , piece_id=FJEQZBYBYMV5UGJNIQQQMTIRGS52FRFOJDIR5GRCSF6J6Q45YJSQ, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.217288910 (323h20m20.679381394s) [noise=true , address=77.90.31.11:4554 , piece_id=TUC6E47W7ZQKBRGS67HFB6GQECOL3477QF3SBKKEXSPQRGODS65A, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.219923928 (323h20m20.682016411s) [noise=true , address=77.90.31.13:4626 , piece_id=VTYR7PC6K63N74BABLN547DFUSKTG6JDCI3TDDMZ3G443Z3T6W5A, seg_pos={0 0}, upload=284560]: Done uploading piece. err: <nil> Oct 7 00:51:35.219934047 (323h20m20.68202653s) [seg_pos={0 0}, upload=284560]: Segment reached optimal threshold of 13 pieces. Oct 7 00:51:35.220097188 (323h20m20.682189661s) [noise=true , address=45.91.116.8:4047 , piece_id=USZEPY64NMNWKP3GNGNUQYKHRYRY6LVU75ND7VMFFU76PPRZNN5Q, seg_pos={0 0}, upload=284560]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Oct 7 00:51:35.220106335 (323h20m20.682198818s) [noise=true , address=45.91.116.8:4047 , piece_id=USZEPY64NMNWKP3GNGNUQYKHRYRY6LVU75ND7VMFFU76PPRZNN5Q, seg_pos={0 0}, upload=284560]: Long tail cancellation due to optimal threshold Oct 7 00:51:35.220185816 (323h20m20.6822783s) [noise=true , address=45.91.116.14:4189 , piece_id=XMVJY3Z3A5GZLGI7LMTAKHS4V3DUS7P5FTJ4EPN3ALMJ4UUX64PA, seg_pos={0 0}, upload=284560]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Oct 7 00:51:35.220191547 (323h20m20.68228403s) [noise=true , address=45.91.116.14:4189 , piece_id=XMVJY3Z3A5GZLGI7LMTAKHS4V3DUS7P5FTJ4EPN3ALMJ4UUX64PA, seg_pos={0 0}, upload=284560]: Long tail cancellation due to optimal threshold Oct 7 00:51:35.220246823 (323h20m20.682339306s) [noise=true , address=45.91.116.11:4118 , piece_id=EUTAWK3JE6ZCZMLGEJKYDEBSN7Q3WKATOEO5BD4CP37QESDFESQQ, seg_pos={0 0}, upload=284560]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Oct 7 00:51:35.220253205 (323h20m20.682345688s) [noise=true , address=45.91.116.11:4118 , piece_id=EUTAWK3JE6ZCZMLGEJKYDEBSN7Q3WKATOEO5BD4CP37QESDFESQQ, seg_pos={0 0}, upload=284560]: Long tail cancellation due to optimal threshold Oct 7 00:51:35.220272602 (323h20m20.682365075s) [seg_pos={0 0}, upload=284560]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:51:35.220355991 (323h20m20.682448464s) [upload=284560]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:51:35.220397750 (323h20m20.682490233s) [upload=284560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:51:35.601020299 (323h20m21.063112782s) [upload=284560]: Done uploading segments... Oct 7 00:51:35.601045828 (323h20m21.063138311s) [upload=284560]: Upload finished. err: <nil> Duration: 30.002579867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 30.001610917s FAIL (2025-10-07 00:51:04 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.00162794s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 25.787861236s FAIL (2025-10-07 00:51:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 25.787916872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 9.783680143s FAIL (2025-10-07 00:50:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 00:48:56.613229313 (323h17m42.075321796s) [upload=284523]: Starting upload Oct 7 00:48:56.613324523 (323h17m42.075417007s) [upload=284523]: Done starting upload Oct 7 00:48:56.613384268 (323h17m42.075476751s) [upload=284523]: Uploading segments... Oct 7 00:48:56.613615168 (323h17m42.075707641s) [upload=284523]: Got next segment. Inline: false Oct 7 00:48:56.613663609 (323h17m42.075756082s) [upload=284523]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:48:56.613669240 (323h17m42.075761723s) [upload=284523]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:48:57.190575036 (323h17m42.652667519s) [seg_pos={0 0}, upload=284523]: Begin upload segment... Oct 7 00:48:57.190638877 (323h17m42.65273136s) [seg_pos={0 0}, upload=284523]: Done begin upload segment. Oct 7 00:48:57.190666610 (323h17m42.652759083s) [noise=true , address=145.79.227.19:4079 , piece_id=N2HVTF7Z5LWHTUU2XP5U7H6OGZE5FPLGXWBM6LUBV4S63OYCKIZQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.190709111 (323h17m42.652801594s) [noise=true , address=145.79.227.16:4909 , piece_id=AF5U7EEX66OXIUGORRI7PARX3GIUHMYWEWV4IQRQG2UBHJVUKNZA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191214002 (323h17m42.653306475s) [noise=true , address=145.79.227.12:4767 , piece_id=FAUAYZLZS3NHJSVOG4G6OSWCIYFMA2V4TYBAKVHXEIEKKTCCVYEA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191500899 (323h17m42.653593372s) [noise=true , address=45.91.116.8:4048 , piece_id=WAWXPIYZT5FJDN3WMDLBYXXVFYG2RQQC6ZY3UJWHVGFOQTOVXPLQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191879570 (323h17m42.653972063s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.192213936 (323h17m42.654306409s) [noise=true , address=145.79.227.14:4837 , piece_id=F4TSTSTHX5BX3OADIIGUIFS26TXCFQPGVS75VLB3HQEABEX6AT7A, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.192648384 (323h17m42.654740867s) [noise=true , address=145.79.227.3:4151 , piece_id=POHBULLPCNQDQYDXHQO5IERYITYGXCYQQMMKJDAN6ZB7BJILYCZQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.193009221 (323h17m42.655101694s) [noise=true , address=145.79.227.18:4005 , piece_id=7Y2NPR5TOSPDZHGK5GWKI7WRW55W7YFQFOWS3Z4A4LSW7RUYU7LA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.193453617 (323h17m42.6555461s) [noise=true , address=145.79.227.4:4186 , piece_id=O6JFC6Q75TSMLYEYMN3LGCSDSCH7IZQZGGR4ZDHN7PNLPXWR46TQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.194729406 (323h17m42.656821889s) [noise=true , address=77.90.31.21:4877 , piece_id=KUWJQN5THELOZ2DCE7WVN23UDCTOZEMOPLRQF36QWKHKZEQBQYKA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.196132077 (323h17m42.658224561s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.198421616 (323h17m42.660514099s) [noise=true , address=45.91.116.13:4155 , piece_id=LQLGG276QYLAFCV7N25H4I25JXR4QXP647OUFIOZG4DZ7EVNFFFA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.200790979 (323h17m42.662883452s) [noise=true , address=145.79.227.2:4113 , piece_id=R3XEJLWII4XLIESE46RJBOYQESBJKC6P7SHXQ64O4UQZITPZMFJQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.202469044 (323h17m42.664561527s) [noise=true , address=145.79.227.16:4911 , piece_id=P6Q6O6VBBY5F4S774JGMH3BXQYW46PRQWWXNOKM4YCTXOLADOBLA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.202952645 (323h17m42.665045128s) [noise=true , address=145.79.227.17:4972 , piece_id=KXVAP72FVEAHTZ6RQH5I5Q3LYKRBUUUZZOKZWOLLCBZHAYCZKTMQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.204311693 (323h17m42.666404176s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:58.852892147 (323h17m44.31498463s) [upload=284523]: Next returned nil segment Oct 7 00:48:58.852911884 (323h17m44.315004367s) [upload=284523]: Waiting for error group managing segments... Oct 7 00:48:59.177446373 (323h17m44.639538846s) [noise=true , address=145.79.227.12:4767 , piece_id=FAUAYZLZS3NHJSVOG4G6OSWCIYFMA2V4TYBAKVHXEIEKKTCCVYEA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.180797364 (323h17m44.642889847s) [noise=true , address=145.79.227.19:4079 , piece_id=N2HVTF7Z5LWHTUU2XP5U7H6OGZE5FPLGXWBM6LUBV4S63OYCKIZQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.185354562 (323h17m44.647447045s) [noise=true , address=145.79.227.18:4005 , piece_id=7Y2NPR5TOSPDZHGK5GWKI7WRW55W7YFQFOWS3Z4A4LSW7RUYU7LA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.185828054 (323h17m44.647920527s) [noise=true , address=145.79.227.4:4186 , piece_id=O6JFC6Q75TSMLYEYMN3LGCSDSCH7IZQZGGR4ZDHN7PNLPXWR46TQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.190734186 (323h17m44.652826659s) [noise=true , address=145.79.227.17:4972 , piece_id=KXVAP72FVEAHTZ6RQH5I5Q3LYKRBUUUZZOKZWOLLCBZHAYCZKTMQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.192115355 (323h17m44.654207838s) [noise=true , address=145.79.227.16:4909 , piece_id=AF5U7EEX66OXIUGORRI7PARX3GIUHMYWEWV4IQRQG2UBHJVUKNZA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.201433238 (323h17m44.663525711s) [noise=true , address=145.79.227.3:4151 , piece_id=POHBULLPCNQDQYDXHQO5IERYITYGXCYQQMMKJDAN6ZB7BJILYCZQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.202680584 (323h17m44.664773057s) [noise=true , address=145.79.227.14:4837 , piece_id=F4TSTSTHX5BX3OADIIGUIFS26TXCFQPGVS75VLB3HQEABEX6AT7A, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.208003650 (323h17m44.670096123s) [noise=true , address=145.79.227.2:4113 , piece_id=R3XEJLWII4XLIESE46RJBOYQESBJKC6P7SHXQ64O4UQZITPZMFJQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.245499177 (323h17m44.70759165s) [noise=true , address=145.79.227.16:4911 , piece_id=P6Q6O6VBBY5F4S774JGMH3BXQYW46PRQWWXNOKM4YCTXOLADOBLA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.337714280 (323h17m44.799806763s) [noise=true , address=45.91.116.13:4155 , piece_id=LQLGG276QYLAFCV7N25H4I25JXR4QXP647OUFIOZG4DZ7EVNFFFA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.359104120 (323h17m44.821196604s) [noise=true , address=45.91.116.8:4048 , piece_id=WAWXPIYZT5FJDN3WMDLBYXXVFYG2RQQC6ZY3UJWHVGFOQTOVXPLQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.360706612 (323h17m44.822799095s) [noise=true , address=77.90.31.21:4877 , piece_id=KUWJQN5THELOZ2DCE7WVN23UDCTOZEMOPLRQF36QWKHKZEQBQYKA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.360777166 (323h17m44.822869659s) [seg_pos={0 0}, upload=284523]: Segment reached optimal threshold of 13 pieces. Oct 7 00:48:59.360932752 (323h17m44.823025225s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.360944375 (323h17m44.823036848s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361053563 (323h17m44.823146036s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.361062389 (323h17m44.823154872s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361107885 (323h17m44.823200359s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.361129166 (323h17m44.823221649s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361162901 (323h17m44.823255374s) [seg_pos={0 0}, upload=284523]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:48:59.361244957 (323h17m44.82333743s) [upload=284523]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:48:59.361250958 (323h17m44.823343431s) [upload=284523]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:48:59.734954893 (323h17m45.197047386s) [upload=284523]: Done uploading segments... Oct 7 00:48:59.734978559 (323h17m45.197071052s) [upload=284523]: Upload finished. err: <nil> Duration: 9.783711585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 19.121502596s FAIL (2025-10-07 00:50:33 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 19.121535799s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 22.75882405s FAIL (2025-10-07 00:50:15 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 22.758848075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.002337474s FAIL (2025-10-07 00:49:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:49:11.270477690 (323h17m56.732570163s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.271660442 (323h17m56.733752925s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272073198 (323h17m56.734165681s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.272607475 (323h17m56.734699968s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273069225 (323h17m56.735161698s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273469988 (323h17m56.735562471s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.273935304 (323h17m56.736027787s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.274462949 (323h17m56.736555432s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.275071137 (323h17m56.73716361s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.277843135 (323h17m56.739935619s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.279164522 (323h17m56.741256995s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.282786378 (323h17m56.744878851s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284392847 (323h17m56.74648533s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.284956089 (323h17m56.747048562s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.286010447 (323h17m56.74810292s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287033115 (323h17m56.749143441s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:11.287555179 (323h17m56.749647662s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Uploading piece... Oct 7 00:49:12.734127766 (323h17m58.19622025s) [upload=284525]: Next returned nil segment Oct 7 00:49:12.734149448 (323h17m58.196241921s) [upload=284525]: Waiting for error group managing segments... Oct 7 00:49:13.140405618 (323h17m58.602498101s) [noise=true , address=145.79.227.23:4566 , piece_id=R7Y4CG2LDCK4MDNV2JHG5Z66CPZFDRK5PZV6FEHYDAWWLLDKIFWQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.140737971 (323h17m58.602830454s) [noise=true , address=145.79.227.16:4919 , piece_id=IVTVRCHBJAMWAO4RB2WYU7JNBEDNF3RCTMPX5BZJ3DWMU43R2Q5A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.147417470 (323h17m58.609509953s) [noise=true , address=145.79.227.31:4813 , piece_id=YZUK53DVVYXC3SQNY2CMZUYJCLIRDF5YK2MKW2H5HNYVNRL33BMA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.159901651 (323h17m58.621994124s) [noise=true , address=145.79.227.27:4694 , piece_id=3DMPF2HRM5AATOTNKOMLHVF2H62M5GNEPZD36PKOB7JOXYSQQLGQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.164472215 (323h17m58.626564698s) [noise=true , address=145.79.227.22:4514 , piece_id=J5S6DVBKN5TSSAC66IYQLNW4IDBQQ7G6QPBU4U723DYUAWCEKRTQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.264687051 (323h17m58.726779534s) [noise=true , address=45.91.116.13:4169 , piece_id=XIJA6BU3BVA3WS3RFNPMMLUXMCFVSLDSQTMQG7INJE6IQAMCKF6Q, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.267171182 (323h17m58.729263655s) [noise=true , address=77.90.31.10:4542 , piece_id=ABVXFVZOWDY6KKNEJMM3AYS4WM5E77JDZOYGWJ43Q4BBCMWEKZNQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.272807604 (323h17m58.734900087s) [noise=true , address=45.91.116.4:4943 , piece_id=DGVZZHGPKN2TQ37TR2UWXEZMLGJLVZEQL3SI5C2W26K7BLUSLDPQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.278449096 (323h17m58.740541569s) [noise=true , address=77.90.31.21:4896 , piece_id=YRZ6T2CRK5NPIE532I23AHVYGHO2MCVC6D42LA72USWW3NZLEZQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.299289319 (323h17m58.761381792s) [noise=true , address=45.91.116.12:4249 , piece_id=F5E65FG3DESXPVPE33377JXUFYRNJMNDGGL6HX77NBVTJPZIVXAA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.311651950 (323h17m58.773744423s) [noise=true , address=45.91.116.5:4303 , piece_id=RXH6G3NNH3HTBDSJ2OKXIBUZBSD6QNRVVODJVLIFXKT4QUWTDRGA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.323896927 (323h17m58.7859894s) [noise=true , address=77.90.31.15:4718 , piece_id=AJOIU3PWFOMNZOROFECJUX2SQYZPNXHNNDGSZ44KTPP4N2FIIIDQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.361671816 (323h17m58.823764299s) [noise=true , address=77.90.31.12:4601 , piece_id=JQOQKYYV6X5M7FL43KCPSED5IF5JK4GUQWHCRJRBVQ22PYQLRNIQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364388519 (323h17m58.826481002s) [noise=true , address=45.91.116.9:4277 , piece_id=K6DCOVH57MA77REORPN6IYYXHSCMUAJ2UHUBEKFBHVSEPPNSM5KQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: <nil> Oct 7 00:49:13.364415841 (323h17m58.826508324s) [seg_pos={0 0}, upload=284525]: Segment reached optimal threshold of 14 pieces. Oct 7 00:49:13.364650007 (323h17m58.82674249s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364659244 (323h17m58.826751727s) [noise=true , address=77.90.31.17:4789 , piece_id=6PTPQ4BQNCB7GHVJTH374W3DCLZWUUS2OBKFFDDY6NLZC5MACK6A, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364690343 (323h17m58.826782847s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364702036 (323h17m58.826794519s) [noise=true , address=45.91.116.8:4068 , piece_id=BVE7LSHTR2AQ24X5HQ6HVMRQIWJVZLGIG5DPVHTJH3VVDGBA2BVQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364734859 (323h17m58.826827352s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UVCFqkZwuXQ28GdfQahRcC5jwfWhoanWdTs7Z1jdHkxVVDARi): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364741732 (323h17m58.826834215s) [noise=true , address=77.90.31.19:4852 , piece_id=JHUSGCTYLD7SMQKFC3CNU6URQMFEZEC53QNWQCL2ERFMEWSQVHSQ, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364793700 (323h17m58.826886173s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq): protocol: expected piece hash; context canceled; EOF Oct 7 00:49:13.364801255 (323h17m58.826893728s) [noise=true , address=77.90.31.11:4570 , piece_id=2S6KTNFM27XDGRGJKUJCAK7AVBUXYYIOIR52DQ5SU4B7GFDEMBQA, seg_pos={0 0}, upload=284525]: Long tail cancellation due to optimal threshold Oct 7 00:49:13.364821233 (323h17m58.826913716s) [seg_pos={0 0}, upload=284525]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 00:49:13.364916545 (323h17m58.827009028s) [upload=284525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:49:13.364924219 (323h17m58.827016702s) [upload=284525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:49:32.774142630 (323h18m18.236235103s) [upload=284525]: Done uploading segments... Oct 7 00:49:32.774167727 (323h18m18.2362602s) [upload=284525]: Upload finished. err: <nil> Duration: 1m0.002373843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 25.262001788s FAIL (2025-10-07 00:49:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 00:48:56.613229313 (323h17m42.075321796s) [upload=284523]: Starting upload Oct 7 00:48:56.613324523 (323h17m42.075417007s) [upload=284523]: Done starting upload Oct 7 00:48:56.613384268 (323h17m42.075476751s) [upload=284523]: Uploading segments... Oct 7 00:48:56.613615168 (323h17m42.075707641s) [upload=284523]: Got next segment. Inline: false Oct 7 00:48:56.613663609 (323h17m42.075756082s) [upload=284523]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:48:56.613669240 (323h17m42.075761723s) [upload=284523]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:48:57.190575036 (323h17m42.652667519s) [seg_pos={0 0}, upload=284523]: Begin upload segment... Oct 7 00:48:57.190638877 (323h17m42.65273136s) [seg_pos={0 0}, upload=284523]: Done begin upload segment. Oct 7 00:48:57.190666610 (323h17m42.652759083s) [noise=true , address=145.79.227.19:4079 , piece_id=N2HVTF7Z5LWHTUU2XP5U7H6OGZE5FPLGXWBM6LUBV4S63OYCKIZQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.190709111 (323h17m42.652801594s) [noise=true , address=145.79.227.16:4909 , piece_id=AF5U7EEX66OXIUGORRI7PARX3GIUHMYWEWV4IQRQG2UBHJVUKNZA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191214002 (323h17m42.653306475s) [noise=true , address=145.79.227.12:4767 , piece_id=FAUAYZLZS3NHJSVOG4G6OSWCIYFMA2V4TYBAKVHXEIEKKTCCVYEA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191500899 (323h17m42.653593372s) [noise=true , address=45.91.116.8:4048 , piece_id=WAWXPIYZT5FJDN3WMDLBYXXVFYG2RQQC6ZY3UJWHVGFOQTOVXPLQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191879570 (323h17m42.653972063s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.192213936 (323h17m42.654306409s) [noise=true , address=145.79.227.14:4837 , piece_id=F4TSTSTHX5BX3OADIIGUIFS26TXCFQPGVS75VLB3HQEABEX6AT7A, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.192648384 (323h17m42.654740867s) [noise=true , address=145.79.227.3:4151 , piece_id=POHBULLPCNQDQYDXHQO5IERYITYGXCYQQMMKJDAN6ZB7BJILYCZQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.193009221 (323h17m42.655101694s) [noise=true , address=145.79.227.18:4005 , piece_id=7Y2NPR5TOSPDZHGK5GWKI7WRW55W7YFQFOWS3Z4A4LSW7RUYU7LA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.193453617 (323h17m42.6555461s) [noise=true , address=145.79.227.4:4186 , piece_id=O6JFC6Q75TSMLYEYMN3LGCSDSCH7IZQZGGR4ZDHN7PNLPXWR46TQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.194729406 (323h17m42.656821889s) [noise=true , address=77.90.31.21:4877 , piece_id=KUWJQN5THELOZ2DCE7WVN23UDCTOZEMOPLRQF36QWKHKZEQBQYKA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.196132077 (323h17m42.658224561s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.198421616 (323h17m42.660514099s) [noise=true , address=45.91.116.13:4155 , piece_id=LQLGG276QYLAFCV7N25H4I25JXR4QXP647OUFIOZG4DZ7EVNFFFA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.200790979 (323h17m42.662883452s) [noise=true , address=145.79.227.2:4113 , piece_id=R3XEJLWII4XLIESE46RJBOYQESBJKC6P7SHXQ64O4UQZITPZMFJQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.202469044 (323h17m42.664561527s) [noise=true , address=145.79.227.16:4911 , piece_id=P6Q6O6VBBY5F4S774JGMH3BXQYW46PRQWWXNOKM4YCTXOLADOBLA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.202952645 (323h17m42.665045128s) [noise=true , address=145.79.227.17:4972 , piece_id=KXVAP72FVEAHTZ6RQH5I5Q3LYKRBUUUZZOKZWOLLCBZHAYCZKTMQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.204311693 (323h17m42.666404176s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:58.852892147 (323h17m44.31498463s) [upload=284523]: Next returned nil segment Oct 7 00:48:58.852911884 (323h17m44.315004367s) [upload=284523]: Waiting for error group managing segments... Oct 7 00:48:59.177446373 (323h17m44.639538846s) [noise=true , address=145.79.227.12:4767 , piece_id=FAUAYZLZS3NHJSVOG4G6OSWCIYFMA2V4TYBAKVHXEIEKKTCCVYEA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.180797364 (323h17m44.642889847s) [noise=true , address=145.79.227.19:4079 , piece_id=N2HVTF7Z5LWHTUU2XP5U7H6OGZE5FPLGXWBM6LUBV4S63OYCKIZQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.185354562 (323h17m44.647447045s) [noise=true , address=145.79.227.18:4005 , piece_id=7Y2NPR5TOSPDZHGK5GWKI7WRW55W7YFQFOWS3Z4A4LSW7RUYU7LA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.185828054 (323h17m44.647920527s) [noise=true , address=145.79.227.4:4186 , piece_id=O6JFC6Q75TSMLYEYMN3LGCSDSCH7IZQZGGR4ZDHN7PNLPXWR46TQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.190734186 (323h17m44.652826659s) [noise=true , address=145.79.227.17:4972 , piece_id=KXVAP72FVEAHTZ6RQH5I5Q3LYKRBUUUZZOKZWOLLCBZHAYCZKTMQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.192115355 (323h17m44.654207838s) [noise=true , address=145.79.227.16:4909 , piece_id=AF5U7EEX66OXIUGORRI7PARX3GIUHMYWEWV4IQRQG2UBHJVUKNZA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.201433238 (323h17m44.663525711s) [noise=true , address=145.79.227.3:4151 , piece_id=POHBULLPCNQDQYDXHQO5IERYITYGXCYQQMMKJDAN6ZB7BJILYCZQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.202680584 (323h17m44.664773057s) [noise=true , address=145.79.227.14:4837 , piece_id=F4TSTSTHX5BX3OADIIGUIFS26TXCFQPGVS75VLB3HQEABEX6AT7A, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.208003650 (323h17m44.670096123s) [noise=true , address=145.79.227.2:4113 , piece_id=R3XEJLWII4XLIESE46RJBOYQESBJKC6P7SHXQ64O4UQZITPZMFJQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.245499177 (323h17m44.70759165s) [noise=true , address=145.79.227.16:4911 , piece_id=P6Q6O6VBBY5F4S774JGMH3BXQYW46PRQWWXNOKM4YCTXOLADOBLA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.337714280 (323h17m44.799806763s) [noise=true , address=45.91.116.13:4155 , piece_id=LQLGG276QYLAFCV7N25H4I25JXR4QXP647OUFIOZG4DZ7EVNFFFA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.359104120 (323h17m44.821196604s) [noise=true , address=45.91.116.8:4048 , piece_id=WAWXPIYZT5FJDN3WMDLBYXXVFYG2RQQC6ZY3UJWHVGFOQTOVXPLQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.360706612 (323h17m44.822799095s) [noise=true , address=77.90.31.21:4877 , piece_id=KUWJQN5THELOZ2DCE7WVN23UDCTOZEMOPLRQF36QWKHKZEQBQYKA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.360777166 (323h17m44.822869659s) [seg_pos={0 0}, upload=284523]: Segment reached optimal threshold of 13 pieces. Oct 7 00:48:59.360932752 (323h17m44.823025225s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.360944375 (323h17m44.823036848s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361053563 (323h17m44.823146036s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.361062389 (323h17m44.823154872s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361107885 (323h17m44.823200359s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.361129166 (323h17m44.823221649s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361162901 (323h17m44.823255374s) [seg_pos={0 0}, upload=284523]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:48:59.361244957 (323h17m44.82333743s) [upload=284523]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:48:59.361250958 (323h17m44.823343431s) [upload=284523]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:48:59.734954893 (323h17m45.197047386s) [upload=284523]: Done uploading segments... Oct 7 00:48:59.734978559 (323h17m45.197071052s) [upload=284523]: Upload finished. err: <nil> Duration: 25.262021866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.004333381s FAIL (2025-10-07 00:49:43 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.004355063s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001887431s FAIL (2025-10-07 00:49:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 7 00:48:56.613229313 (323h17m42.075321796s) [upload=284523]: Starting upload Oct 7 00:48:56.613324523 (323h17m42.075417007s) [upload=284523]: Done starting upload Oct 7 00:48:56.613384268 (323h17m42.075476751s) [upload=284523]: Uploading segments... Oct 7 00:48:56.613615168 (323h17m42.075707641s) [upload=284523]: Got next segment. Inline: false Oct 7 00:48:56.613663609 (323h17m42.075756082s) [upload=284523]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 00:48:56.613669240 (323h17m42.075761723s) [upload=284523]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 00:48:57.190575036 (323h17m42.652667519s) [seg_pos={0 0}, upload=284523]: Begin upload segment... Oct 7 00:48:57.190638877 (323h17m42.65273136s) [seg_pos={0 0}, upload=284523]: Done begin upload segment. Oct 7 00:48:57.190666610 (323h17m42.652759083s) [noise=true , address=145.79.227.19:4079 , piece_id=N2HVTF7Z5LWHTUU2XP5U7H6OGZE5FPLGXWBM6LUBV4S63OYCKIZQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.190709111 (323h17m42.652801594s) [noise=true , address=145.79.227.16:4909 , piece_id=AF5U7EEX66OXIUGORRI7PARX3GIUHMYWEWV4IQRQG2UBHJVUKNZA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191214002 (323h17m42.653306475s) [noise=true , address=145.79.227.12:4767 , piece_id=FAUAYZLZS3NHJSVOG4G6OSWCIYFMA2V4TYBAKVHXEIEKKTCCVYEA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191500899 (323h17m42.653593372s) [noise=true , address=45.91.116.8:4048 , piece_id=WAWXPIYZT5FJDN3WMDLBYXXVFYG2RQQC6ZY3UJWHVGFOQTOVXPLQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.191879570 (323h17m42.653972063s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.192213936 (323h17m42.654306409s) [noise=true , address=145.79.227.14:4837 , piece_id=F4TSTSTHX5BX3OADIIGUIFS26TXCFQPGVS75VLB3HQEABEX6AT7A, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.192648384 (323h17m42.654740867s) [noise=true , address=145.79.227.3:4151 , piece_id=POHBULLPCNQDQYDXHQO5IERYITYGXCYQQMMKJDAN6ZB7BJILYCZQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.193009221 (323h17m42.655101694s) [noise=true , address=145.79.227.18:4005 , piece_id=7Y2NPR5TOSPDZHGK5GWKI7WRW55W7YFQFOWS3Z4A4LSW7RUYU7LA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.193453617 (323h17m42.6555461s) [noise=true , address=145.79.227.4:4186 , piece_id=O6JFC6Q75TSMLYEYMN3LGCSDSCH7IZQZGGR4ZDHN7PNLPXWR46TQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.194729406 (323h17m42.656821889s) [noise=true , address=77.90.31.21:4877 , piece_id=KUWJQN5THELOZ2DCE7WVN23UDCTOZEMOPLRQF36QWKHKZEQBQYKA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.196132077 (323h17m42.658224561s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.198421616 (323h17m42.660514099s) [noise=true , address=45.91.116.13:4155 , piece_id=LQLGG276QYLAFCV7N25H4I25JXR4QXP647OUFIOZG4DZ7EVNFFFA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.200790979 (323h17m42.662883452s) [noise=true , address=145.79.227.2:4113 , piece_id=R3XEJLWII4XLIESE46RJBOYQESBJKC6P7SHXQ64O4UQZITPZMFJQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.202469044 (323h17m42.664561527s) [noise=true , address=145.79.227.16:4911 , piece_id=P6Q6O6VBBY5F4S774JGMH3BXQYW46PRQWWXNOKM4YCTXOLADOBLA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.202952645 (323h17m42.665045128s) [noise=true , address=145.79.227.17:4972 , piece_id=KXVAP72FVEAHTZ6RQH5I5Q3LYKRBUUUZZOKZWOLLCBZHAYCZKTMQ, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:57.204311693 (323h17m42.666404176s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Uploading piece... Oct 7 00:48:58.852892147 (323h17m44.31498463s) [upload=284523]: Next returned nil segment Oct 7 00:48:58.852911884 (323h17m44.315004367s) [upload=284523]: Waiting for error group managing segments... Oct 7 00:48:59.177446373 (323h17m44.639538846s) [noise=true , address=145.79.227.12:4767 , piece_id=FAUAYZLZS3NHJSVOG4G6OSWCIYFMA2V4TYBAKVHXEIEKKTCCVYEA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.180797364 (323h17m44.642889847s) [noise=true , address=145.79.227.19:4079 , piece_id=N2HVTF7Z5LWHTUU2XP5U7H6OGZE5FPLGXWBM6LUBV4S63OYCKIZQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.185354562 (323h17m44.647447045s) [noise=true , address=145.79.227.18:4005 , piece_id=7Y2NPR5TOSPDZHGK5GWKI7WRW55W7YFQFOWS3Z4A4LSW7RUYU7LA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.185828054 (323h17m44.647920527s) [noise=true , address=145.79.227.4:4186 , piece_id=O6JFC6Q75TSMLYEYMN3LGCSDSCH7IZQZGGR4ZDHN7PNLPXWR46TQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.190734186 (323h17m44.652826659s) [noise=true , address=145.79.227.17:4972 , piece_id=KXVAP72FVEAHTZ6RQH5I5Q3LYKRBUUUZZOKZWOLLCBZHAYCZKTMQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.192115355 (323h17m44.654207838s) [noise=true , address=145.79.227.16:4909 , piece_id=AF5U7EEX66OXIUGORRI7PARX3GIUHMYWEWV4IQRQG2UBHJVUKNZA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.201433238 (323h17m44.663525711s) [noise=true , address=145.79.227.3:4151 , piece_id=POHBULLPCNQDQYDXHQO5IERYITYGXCYQQMMKJDAN6ZB7BJILYCZQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.202680584 (323h17m44.664773057s) [noise=true , address=145.79.227.14:4837 , piece_id=F4TSTSTHX5BX3OADIIGUIFS26TXCFQPGVS75VLB3HQEABEX6AT7A, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.208003650 (323h17m44.670096123s) [noise=true , address=145.79.227.2:4113 , piece_id=R3XEJLWII4XLIESE46RJBOYQESBJKC6P7SHXQ64O4UQZITPZMFJQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.245499177 (323h17m44.70759165s) [noise=true , address=145.79.227.16:4911 , piece_id=P6Q6O6VBBY5F4S774JGMH3BXQYW46PRQWWXNOKM4YCTXOLADOBLA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.337714280 (323h17m44.799806763s) [noise=true , address=45.91.116.13:4155 , piece_id=LQLGG276QYLAFCV7N25H4I25JXR4QXP647OUFIOZG4DZ7EVNFFFA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.359104120 (323h17m44.821196604s) [noise=true , address=45.91.116.8:4048 , piece_id=WAWXPIYZT5FJDN3WMDLBYXXVFYG2RQQC6ZY3UJWHVGFOQTOVXPLQ, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.360706612 (323h17m44.822799095s) [noise=true , address=77.90.31.21:4877 , piece_id=KUWJQN5THELOZ2DCE7WVN23UDCTOZEMOPLRQF36QWKHKZEQBQYKA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: <nil> Oct 7 00:48:59.360777166 (323h17m44.822869659s) [seg_pos={0 0}, upload=284523]: Segment reached optimal threshold of 13 pieces. Oct 7 00:48:59.360932752 (323h17m44.823025225s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.360944375 (323h17m44.823036848s) [noise=true , address=45.91.116.5:4298 , piece_id=I4Y2VJ35D4JAVHHBEHXJHTMLRJVM5TVCCWCZA6YKL6IUTWZ4NIBA, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361053563 (323h17m44.823146036s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.361062389 (323h17m44.823154872s) [noise=true , address=77.90.31.19:4841 , piece_id=KRXSSKQMWLFQ673FICAEFN3OPJU5332OZU6E5OAWHNL72AQECD2A, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361107885 (323h17m44.823200359s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Oct 7 00:48:59.361129166 (323h17m44.823221649s) [noise=true , address=45.91.116.10:4083 , piece_id=UN5CEPX4HUVO7RW52SPMJEMA6J5DRHOL6WXQCCNOMEMGQUBLUY4Q, seg_pos={0 0}, upload=284523]: Long tail cancellation due to optimal threshold Oct 7 00:48:59.361162901 (323h17m44.823255374s) [seg_pos={0 0}, upload=284523]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 00:48:59.361244957 (323h17m44.82333743s) [upload=284523]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 00:48:59.361250958 (323h17m44.823343431s) [upload=284523]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 00:48:59.734954893 (323h17m45.197047386s) [upload=284523]: Done uploading segments... Oct 7 00:48:59.734978559 (323h17m45.197071052s) [upload=284523]: Upload finished. err: <nil> Duration: 30.001911046s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.165350918s FAIL (2025-10-07 00:01:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.16539977s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.512666177s FAIL (2025-10-07 00:01:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.512695311s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.279979371s FAIL (2025-10-07 00:00:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.280001374s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.276585608s FAIL (2025-10-06 23:59:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments... Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment... Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment. Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments... Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments... Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.276747886s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.160635238s FAIL (2025-10-06 23:59:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.160661518s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.162861237s FAIL (2025-10-06 23:59:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments... Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment... Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment. Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments... Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments... Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.162897245s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.362791537s FAIL (2025-10-06 23:58:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.362819402s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.292404762s FAIL (2025-10-06 23:58:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments... Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment... Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment. Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments... Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments... Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.292440981s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.371707296s FAIL (2025-10-06 23:58:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.371762661s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.....X.....................................................] duration: 6.222178989s FAIL (2025-10-06 23:58:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.222332281s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.....X....................................................] duration: 6.309740451s FAIL (2025-10-06 23:57:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.309794383s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.853247889s FAIL (2025-10-06 23:57:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments... Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment... Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment. Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments... Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments... Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.853278337s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.057910072s FAIL (2025-10-06 23:57:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments... Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment... Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment. Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments... Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments... Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.05794532s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.....X...................................................] duration: 6.10629278s FAIL (2025-10-06 23:56:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.10634534s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X....X......................................................] duration: 6.177333254s FAIL (2025-10-06 23:56:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments... Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment... Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment. Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments... Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments... Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.177361669s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.....X..................................................] duration: 6.176283134s FAIL (2025-10-06 23:56:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.17630705s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.000530401s FAIL (2025-10-06 23:56:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment... Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment. Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments... Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments... Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 6.000561932s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.....X.................................................] duration: 6.497116346s FAIL (2025-10-06 23:56:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.497144069s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X....X.....................................................] duration: 6.098285433s FAIL (2025-10-06 23:55:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments... Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment... Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment. Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments... Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments... Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.098313026s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.710252492s FAIL (2025-10-06 23:55:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.710276508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X....X....................................................] duration: 6.135460202s FAIL (2025-10-06 23:55:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments... Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment... Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment. Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments... Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments... Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.135491823s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X....X...................................................] duration: 6.172812104s FAIL (2025-10-06 23:55:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments... Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment... Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment. Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments... Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments... Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.172847754s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.857911201s FAIL (2025-10-06 23:55:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment... Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment. Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments... Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments... Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.857964493s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.....X................................................] duration: 6.147355823s FAIL (2025-10-06 23:54:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.147382424s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.635829167s FAIL (2025-10-06 23:54:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.635854197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...X....X..................................................] duration: 6.047990694s FAIL (2025-10-06 23:54:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments... Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment... Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment. Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments... Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments... Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.048021442s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.892721604s FAIL (2025-10-06 23:54:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.892750218s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.211212578s FAIL (2025-10-06 23:54:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment... Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment. Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments... Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments... Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 6.211238626s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.....X...............................................] duration: 6.235727177s FAIL (2025-10-06 23:53:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.235755491s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X...X....X.................................................] duration: 6.11382122s FAIL (2025-10-06 23:53:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments... Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment... Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment. Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments... Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments... Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.113866466s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..X........................................................] duration: 5.797116826s FAIL (2025-10-06 23:53:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment... Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment. Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments... Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments... Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.797142046s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.....X..............................................] duration: 6.228097549s FAIL (2025-10-06 23:53:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.228126945s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.....X.............................................] duration: 6.173620475s FAIL (2025-10-06 23:53:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.173657345s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration: 5.816416709s FAIL (2025-10-06 23:53:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment... Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment. Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments... Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments... Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.816452958s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X...X....X................................................] duration: 6.069028894s FAIL (2025-10-06 23:53:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments... Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment... Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment. Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments... Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments... Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.069060053s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.814037127s FAIL (2025-10-06 23:53:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.81436999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..X......................................................] duration: 5.985760699s FAIL (2025-10-06 23:52:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:14:43.567648320 (321h43m29.029740793s) [seg_pos={0 0}, upload=283145]: Begin upload segment... Oct 6 23:14:43.567694077 (321h43m29.02978656s) [seg_pos={0 0}, upload=283145]: Done begin upload segment. Oct 6 23:14:43.567718734 (321h43m29.029811217s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.569940475 (321h43m29.032032958s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570021860 (321h43m29.032114343s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570439425 (321h43m29.032531898s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570486445 (321h43m29.032578938s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570742031 (321h43m29.032834514s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.570861739 (321h43m29.032954212s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571029488 (321h43m29.033121961s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571262993 (321h43m29.033355476s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571309471 (321h43m29.033401955s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571628941 (321h43m29.033721414s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.571669538 (321h43m29.033762021s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572009554 (321h43m29.034102027s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572029833 (321h43m29.034122317s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572412192 (321h43m29.034504675s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.572800310 (321h43m29.034892804s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573215621 (321h43m29.035308104s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:43.573647284 (321h43m29.035739767s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Uploading piece... Oct 6 23:14:44.938427694 (321h43m30.400520177s) [upload=283145]: Next returned nil segment Oct 6 23:14:44.938451790 (321h43m30.400544263s) [upload=283145]: Waiting for error group managing segments... Oct 6 23:14:45.305487931 (321h43m30.767580404s) [noise=true , address=64.49.13.154:4927 , piece_id=4ZCINADZLFBFVVMGX2J5RZHP2ELHD533PNOFW5KCRJA72NQIJFXQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.414969915 (321h43m30.877062388s) [noise=true , address=145.79.227.38:4977 , piece_id=OR5I5HJH652RQA2QSBIILXUYK645PEGFS2M2AXBHBZEQO2USRBSQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428302803 (321h43m30.890395276s) [noise=true , address=145.79.227.40:4084 , piece_id=C3XIXSNSCV7TZ5A3T3OACJA3XTYIVA32U5J5GUCZCTOB6VAP537Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.428903306 (321h43m30.890995789s) [noise=true , address=145.79.227.33:4869 , piece_id=V2LUMOB6WPNOYVMC7TFKRE2MJQBC4ZM47DP2NUWZOUI5QF22LYCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.431178959 (321h43m30.893271432s) [noise=true , address=145.79.227.22:4526 , piece_id=ZK7R5WEK7UFCHEYMTNHIK74CBNHOUDWN6XKAYPFVI2BDSIDE7KPQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.435550784 (321h43m30.897643267s) [noise=true , address=145.79.227.24:4595 , piece_id=ZW4WY7VKW664COPSN3RJH43INVMOVIEH2UI7IJECCGYQ7HPYQ3HA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.470954799 (321h43m30.933047282s) [noise=true , address=145.79.227.23:4546 , piece_id=YIMSTNC4UCLXITG3QJT2JQIBEINXAOTRV7X4X2THX5O4IABYMOCA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:14:45.498774102 (321h43m30.960866575s) [noise=true , address=145.79.227.25:4626 , piece_id=LD52ZDMCZL4M5FVETREU6UD6REEBZ445KPX3YZM2QA2N5BCCUSSA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:36.611266951 (321h44m22.073359424s) [noise=true , address=77.90.31.12:4608 , piece_id=TSMHBEK5F4GXUSFBFN5RPGT3WPUWM7AB3Q5XPR67ZG5W3ZKQPQJQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:37.973205531 (321h44m23.435298004s) [noise=true , address=77.90.31.11:4559 , piece_id=JPU3APHMO5CYE6FCT27ZVSXGP4L32V57FHVZL4IILY7LTNSMAVIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.127571842 (321h44m23.589664325s) [noise=true , address=77.90.31.12:4623 , piece_id=ULNYTPHIL4YZBV6UHUKTVZWKUWPWN5CNROZGYWQTSPEHYFCCUFIQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:38.333411137 (321h44m23.79550362s) [noise=true , address=77.90.31.19:4875 , piece_id=NBW35XP4QXSBDFCVVZZOTHLA46ERDPEVR5SIAE2AXXT4PSGFXFEA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: <nil> Oct 6 23:15:42.578200250 (321h44m28.040292733s) [noise=true , address=77.90.31.18:4820 , piece_id=VKEWDMHFIPAZSCLP7UN6VILIQFVTZURA4P4RN2LAVUCHGYSS7JVA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, address:77.90.31.18:4820): protocol: read tcp 188.166.225.170:47978->77.90.31.18:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578248211 (321h44m28.040340704s) [noise=true , address=77.90.31.14:4691 , piece_id=M2MIBHJHSXKG3OE6F4PI2JTYVBAKNDXARAI7DYWKNEIAICDTBBHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, address:77.90.31.14:4691): protocol: rpc: dial tcp 77.90.31.14:4691: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578389359 (321h44m28.040481852s) [noise=true , address=45.91.116.6:4977 , piece_id=CRU7XUOYVFKXMH44XR6BMMWOJUXIDCR36FLZ2PLJVAQJ7XOGDFMQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1mvA3W6fMbdqveWrRdK4H5uVbcn3xFza8J5YES8V9UZDBQErBY, address:45.91.116.6:4977): protocol: rpc: dial tcp 45.91.116.6:4977: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578410510 (321h44m28.040502993s) [noise=true , address=45.91.116.10:4105 , piece_id=LQVE5KD2LSOJ6UWMNHB65EW3NF6TZRT2JU6NWIMG6WIFJEMJAU2Q, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV, address:45.91.116.10:4105): protocol: rpc: dial tcp 45.91.116.10:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578546970 (321h44m28.040639443s) [noise=true , address=45.91.116.8:4063 , piece_id=M7B5G3PQYUARVSTE2HEXER5A6IFT6RO42PIOQNN6GK36ZOHLNRHA, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm, address:45.91.116.8:4063): protocol: write tcp 188.166.225.170:39886->45.91.116.8:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578588980 (321h44m28.040681453s) [noise=true , address=77.90.31.16:4741 , piece_id=ZOF2W337OGOSVTUIEEYIWJTVISXVUSVHT5Z47CHUABDJQJM5NKHQ, seg_pos={0 0}, upload=283145]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: read tcp 188.166.225.170:35928->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:15:42.578615671 (321h44m28.040708154s) [seg_pos={0 0}, upload=283145]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Oct 6 23:15:42.828664257 (321h44m28.29075673s) [upload=283145]: Done uploading segments... Oct 6 23:15:42.828691899 (321h44m28.290784372s) [upload=283145]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.985793362s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.62453222s FAIL (2025-10-06 23:52:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.624560163s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..X...X....X...............................................] duration: 6.137878186s FAIL (2025-10-06 23:52:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Oct 6 23:19:05.622889818 (321h47m51.084982291s) [upload=283206]: Starting upload Oct 6 23:19:05.623007110 (321h47m51.085099584s) [upload=283206]: Done starting upload Oct 6 23:19:05.623124484 (321h47m51.085216967s) [upload=283206]: Uploading segments... Oct 6 23:19:05.623270602 (321h47m51.085363085s) [upload=283206]: Got next segment. Inline: false Oct 6 23:19:05.623318153 (321h47m51.085410636s) [upload=283206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 23:19:05.623324946 (321h47m51.085417439s) [upload=283206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 23:19:05.872384869 (321h47m51.334477362s) [seg_pos={0 0}, upload=283206]: Begin upload segment... Oct 6 23:19:05.872475001 (321h47m51.334567484s) [seg_pos={0 0}, upload=283206]: Done begin upload segment. Oct 6 23:19:05.872512532 (321h47m51.334605015s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.872551016 (321h47m51.334643499s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873269684 (321h47m51.335362167s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873638165 (321h47m51.335730649s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.873781809 (321h47m51.335874282s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874164267 (321h47m51.33625675s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874618432 (321h47m51.336710915s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.874671664 (321h47m51.336764147s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875051517 (321h47m51.33714401s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875494420 (321h47m51.337586903s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.875926603 (321h47m51.338019086s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.876451433 (321h47m51.338543916s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877083095 (321h47m51.339175588s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877438452 (321h47m51.339530935s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.877972359 (321h47m51.340064852s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:05.878373783 (321h47m51.340466266s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Uploading piece... Oct 6 23:19:07.559393990 (321h47m53.021486473s) [upload=283206]: Next returned nil segment Oct 6 23:19:07.559422053 (321h47m53.021514526s) [upload=283206]: Waiting for error group managing segments... Oct 6 23:19:07.835660821 (321h47m53.297753294s) [noise=true , address=145.79.227.6:4259 , piece_id=6ZIMWZTNKSLB4QUVJC7HVK5MKV2DPL5Y6CS3CMMBK45MF6CZV7SQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.855918876 (321h47m53.318011379s) [noise=true , address=145.79.227.3:4151 , piece_id=OO4I6FD44MXHUUVJV4G4QUYPPY2GXAZ6J7ILFIMFOTGLFPX3KFXQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.856394722 (321h47m53.318487195s) [noise=true , address=145.79.227.18:4006 , piece_id=22JFJFY4VEXM2IAASJI4XJZQYBVOXTG67ERR4TZ56BOQATX2TI6Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861371970 (321h47m53.323464453s) [noise=true , address=145.79.227.16:4909 , piece_id=KOR3PQLCGN7WK4T3ULKG4AJE65YDNN3HNDTBICSJSFS6PI55I7DQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.861953596 (321h47m53.32404609s) [noise=true , address=145.79.227.7:4293 , piece_id=MU3FPEXK3IO7RFYE3Y65QYVOZBYVGW7VKZIPI33PJW5D5Q4X7ZUQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.866812529 (321h47m53.328905012s) [noise=true , address=145.79.227.3:4150 , piece_id=YSV5V73NHXSD5OI7JERCF5CB2DMSUNTFSP62ZSCSOVZGZ6UU35LQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.869069958 (321h47m53.331162441s) [noise=true , address=145.79.227.2:4113 , piece_id=Z2TX2HF2YYNVJU627EE6FMHSX2D7HGXKK7F27J7BCUPMSLTDYZ3Q, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.870374301 (321h47m53.332466784s) [noise=true , address=145.79.227.13:4803 , piece_id=FOIWYWDGGBLDTNG56NSQZSZTSKUM3FSZ4F73HU3BZALK5XYYLSIA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:07.872849204 (321h47m53.334941697s) [noise=true , address=145.79.227.6:4257 , piece_id=ZPT4BK7WG4GD5OSZYVYRU5JEIL7THM3Z7ZSDIGDO3QTCO7MLJRGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: <nil> Oct 6 23:19:34.922271737 (321h48m20.38436422s) [noise=true , address=45.91.116.6:4973 , piece_id=FIETSOFDH2QYL2JZNNBIUZYMBVINZQDVTU5HG4VKENDF4IV52XYA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922430029 (321h48m20.384522532s) [noise=true , address=77.90.31.14:4663 , piece_id=GPR44GP6KEWPIM3QMKZITG6MEPQZH7VVBXPABDGQDKGD6DBIFUNQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922471778 (321h48m20.384564261s) [noise=true , address=77.90.31.19:4843 , piece_id=4L5EBLXXK2HQFO2D7VD4KMOZ5L4QK3OPLEFWNTJU4ITNEPMGNENA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:40480->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922604792 (321h48m20.384697275s) [noise=true , address=77.90.31.10:4517 , piece_id=Y4QT6KHY3XK5LIDEH7QBQ42H2HS2MMOLVWQV32KXMCDLX5Z4OLAQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 188.166.225.170:48868->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922640319 (321h48m20.384732792s) [noise=true , address=45.91.116.7:4009 , piece_id=O7PL25A7IK47A46X5VMWNJBFY7JBCY73MDYBPRSKUWEJIBGLAVGQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: rpc: dial tcp 45.91.116.7:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922684824 (321h48m20.384777307s) [noise=true , address=77.90.31.19:4842 , piece_id=OKNYTVCQDH44MRCQM4CDCX7K7VI2XN5T32ECNYZGMEMFPTTJHEKA, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 188.166.225.170:43822->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 23:19:34.922807137 (321h48m20.38489962s) [noise=true , address=77.90.31.18:4806 , piece_id=PAWZ34LR7V5ZP7CX3MO5TSAA2ECPMK5UDJWMDRGWDAH7S3PJAGBQ, seg_pos={0 0}, upload=283206]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: context deadline exceeded; context deadline exceeded Oct 6 23:19:34.922895054 (321h48m20.384987537s) [seg_pos={0 0}, upload=283206]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Oct 6 23:19:35.164435880 (321h48m20.626528373s) [upload=283206]: Done uploading segments... Oct 6 23:19:35.164469474 (321h48m20.626561957s) [upload=283206]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.13791155s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)