Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X......X......................................] duration: 5.842459579s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 19:06:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.842482854s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X......X.....................................] duration: 7.064163589s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 19:06:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 7.064191441s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X......X....................................] duration: 5.816598994s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 19:05:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.81662292s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X......X...................................] duration: 5.751195949s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 19:04:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.751223842s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X......X..................................] duration: 5.740267796s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 19:04:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.740291872s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X......X.................................] duration: 5.916956966s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 19:03:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.916988335s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X......X................................] duration: 5.990144737s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 19:02:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.990172782s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X......X...............................] duration: 5.859785202s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 19:01:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.85982086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X......X..............................] duration: 5.840893796s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 19:00:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.840924636s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X......X.............................] duration: 5.790421147s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:59:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.790437197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X......X............................] duration: 5.905495902s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:58:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.905512153s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X......X...........................] duration: 5.96023899s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:58:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.96026996s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X......X..........................] duration: 5.984577446s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:58:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.984707205s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X......X.........................] duration: 5.843771796s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:57:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.843803606s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X......X........................] duration: 5.870512136s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:56:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.870537454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X......X.......................] duration: 5.876300619s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:55:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.876330846s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X......X......................] duration: 5.871647739s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:54:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.871674781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X......X.....................] duration: 5.94814892s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:53:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.948188274s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X......X....................] duration: 5.723960003s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:53:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.723989359s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X......X...................] duration: 5.971881993s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:52:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.971912041s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X......X..................] duration: 5.917910157s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:52:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.917947228s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X......X.................] duration: 5.671167331s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:51:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.671193802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X......X................] duration: 5.890494652s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:51:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.890517705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X......X...............] duration: 5.922252462s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:50:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.922270106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X......X..............] duration: 6.136531647s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:49:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 6.136563548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X......X.............] duration: 5.881339877s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:49:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.881375866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X......X............] duration: 5.778075695s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:48:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.77810972s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X......X...........] duration: 6.10928145s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:47:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 6.109310864s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X......X..........] duration: 6.236001989s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:46:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 6.236026846s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X......X.........] duration: 5.935029576s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:45:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.935055614s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X......X........] duration: 5.893796467s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:45:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.893827205s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X......X.......] duration: 5.946127345s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:44:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.946156881s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X......X......] duration: 6.755913073s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:43:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 6.755938983s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X......X.....] duration: 5.890579601s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:42:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.890608024s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X......X....] duration: 5.886515944s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:41:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.886562122s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X......X...] duration: 6.341005813s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:40:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 6.341042674s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X......X..] duration: 6.16871891s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:39:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 6.168738517s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X......X.] duration: 5.693932469s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:37:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 5.693961104s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X......X] duration: 30.003269928s trace: https://jaeger.dev.storj.io/trace/a56ce866745b60a FAIL (2025-11-22 18:37:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ents... Nov 22 18:36:59.221841790 (322h32m57.773312261s) [upload=284055]: Got next segment. Inline: false Nov 22 18:36:59.221903147 (322h32m57.773373619s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 18:36:59.221910531 (322h32m57.773381002s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:36:59.466809533 (322h32m58.018279994s) [seg_pos={0 0}, upload=284055]: Begin upload segment... Nov 22 18:36:59.466882772 (322h32m58.018353234s) [seg_pos={0 0}, upload=284055]: Done begin upload segment. Nov 22 18:36:59.466939671 (322h32m58.018410132s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.466974487 (322h32m58.018444949s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.467670002 (322h32m58.019140463s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468038534 (322h32m58.019508995s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468331692 (322h32m58.019802163s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468351791 (322h32m58.019822252s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.468710163 (322h32m58.020180624s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469106757 (322h32m58.020577218s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469443609 (322h32m58.02091407s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.469558217 (322h32m58.021028678s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470001562 (322h32m58.021472013s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.470276846 (322h32m58.021747297s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.471329480 (322h32m58.022799941s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.472711221 (322h32m58.024181682s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.474113171 (322h32m58.025583632s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:36:59.475455036 (322h32m58.026925497s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Uploading piece... Nov 22 18:37:01.297295712 (322h32m59.848766173s) [upload=284055]: Next returned nil segment Nov 22 18:37:01.297315809 (322h32m59.848786271s) [upload=284055]: Waiting for error group managing segments... Nov 22 18:37:01.598666546 (322h33m0.150137018s) [noise=true , address=77.90.31.14:4663 , piece_id=VY6QN6BVUTQTMNVJ44KCCBNM4HOCWPSBHN6RQUHKDDYSTS6RMPDA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.716002204 (322h33m0.267472665s) [noise=true , address=45.91.116.4:4940 , piece_id=X6ALZFPTZFNFFM6VYF6DXDTGJWUIFJC5PNYFYYNDFMJVJSSLANWQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.718987429 (322h33m0.27045789s) [noise=true , address=77.90.31.15:4697 , piece_id=EBE6WK7G2B5GWSNRJF7HTUKUDO6TXWZEMBRYJQFVNVFPX634BHXQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.749679692 (322h33m0.301150154s) [noise=true , address=45.91.116.5:4298 , piece_id=HELF6VFXP5APZFZ6RGDS3ILS72ZOAS5LYACMAXELI5Z2WZ6CPETA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:01.826391857 (322h33m0.377862318s) [noise=true , address=77.90.31.21:4877 , piece_id=KZIQAV5LN4FBZ22U6RFI7JUNC2O23QZKZ22P6BW4MT56UU7HSZZQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.205688818 (322h33m1.757159279s) [noise=true , address=145.79.227.26:4653 , piece_id=OSCJOETC5LJZ4HOG2RE33I5SBFVNP2RZ4ECA34GJPX34AFUGLV3Q, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:03.596814482 (322h33m2.148284953s) [noise=true , address=145.79.227.21:4475 , piece_id=FC5QUPLLDRPOIFYPAMWVP75H3R4CEXNH27X2E2HGMM3YAD5BJ4RA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:10.907158944 (322h33m9.458629405s) [noise=true , address=145.79.227.32:4834 , piece_id=XPNQBP2UIQA6PBTPZDCXPUJOIOUT2AWDJD3SDIJ3EX2FBWQRRG5A, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:13.679446121 (322h33m12.230916582s) [noise=true , address=145.79.227.8:4330 , piece_id=NEZH73Y7PE55326N6KEV73QVGPKK6UV4EJQO4NR5GUZMIB6A3BKQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:18.388399622 (322h33m16.939870083s) [noise=true , address=145.79.227.13:4804 , piece_id=K6DGS462O3XROWD4OB62GTTXU4K2YG6PIMNEMVXFDTDEHL5J4ZLQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.773670028 (322h33m18.325140489s) [noise=true , address=145.79.227.7:4295 , piece_id=TLMFHM5CEUUAJMM3P6NFO2XPVVCUJ55URWUOTFHRFXNM76ZNTVVA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:19.930963714 (322h33m18.482434165s) [noise=true , address=145.79.227.22:4511 , piece_id=2PAYA7TNHV2TWKXVQ2IICIM6DNH42D4QM247MHTBHKR2DXEI4HTQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038887730 (322h33m24.590358181s) [noise=true , address=145.79.227.15:4876 , piece_id=QZ55ACAU6HKEG6BHJSKSBPRFZ6W2BS6VDFOAQFCQW4ITMMKLL2LQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: <nil> Nov 22 18:37:26.038906827 (322h33m24.590377288s) [seg_pos={0 0}, upload=284055]: Segment reached optimal threshold of 13 pieces. Nov 22 18:37:26.039109202 (322h33m24.590579663s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Nov 22 18:37:26.039117919 (322h33m24.59058838s) [noise=true , address=145.79.227.11:4730 , piece_id=TSAKH3CIZIRNXYYRQZAQ6I4I6PRVAZE7BSYTENZ3KQNNPV5BKONA, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039334882 (322h33m24.590805333s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH): protocol: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57292->145.79.227.21:4474: use of closed network connection Nov 22 18:37:26.039342607 (322h33m24.590813068s) [noise=true , address=145.79.227.21:4474 , piece_id=AJXXP4PA4E3TIYTYVS6XDXYAXHEYRMAPTQ66J44V3A3LCHTUXLJQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039381401 (322h33m24.590851852s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34094->145.79.227.25:4617: use of closed network connection Nov 22 18:37:26.039419844 (322h33m24.590890305s) [noise=true , address=145.79.227.25:4617 , piece_id=BLMZVLWYOUYYPN5STOX4TA55E6WFBPJNOFDINBPFHB77TO5ZDTSQ, seg_pos={0 0}, upload=284055]: Long tail cancellation due to optimal threshold Nov 22 18:37:26.039446996 (322h33m24.590917457s) [seg_pos={0 0}, upload=284055]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 18:37:26.039660462 (322h33m24.591130923s) [upload=284055]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 18:37:26.039669429 (322h33m24.59113989s) [upload=284055]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 18:37:26.337505062 (322h33m24.888975523s) [upload=284055]: Done uploading segments... Nov 22 18:37:26.337541492 (322h33m24.889011953s) [upload=284055]: Upload finished. err: <nil> Duration: 30.00329677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 23.943673055s trace: https://jaeger.dev.storj.io/trace/5225b8d0a62143ca FAIL (2025-11-22 18:36:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== s) [upload=283964]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:30:36.531047319 (322h26m35.08251778s) [seg_pos={0 0}, upload=283964]: Begin upload segment... Nov 22 18:30:36.531092805 (322h26m35.082563256s) [seg_pos={0 0}, upload=283964]: Done begin upload segment. Nov 22 18:30:36.531122301 (322h26m35.082592763s) [noise=true , address=77.90.31.13:4627 , piece_id=VJFT5OTJEZQYSPRL4QHAWFWF7HCRYMUZDESZO4BJM4NW5XZL7ZQQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.532650812 (322h26m35.084121263s) [noise=true , address=145.79.227.17:4970 , piece_id=7JXIJJROS4CCTDQOTOOL6XCNBA3ASBVGCO4YF75G5QRJ2QY2QYAA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.533640798 (322h26m35.085111259s) [noise=true , address=145.79.227.15:4875 , piece_id=HNFPLSNQQQM5M5I7J44WCAQL2XLRQ4GM44IHI7XI2ESNDCPFDWIQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.534514271 (322h26m35.085984732s) [noise=true , address=45.91.116.4:4940 , piece_id=6LWBQSRMQYU7U3TUGUUMPDIOSX6AWCIO3RE46BXRS73YKR7E25DA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.535280560 (322h26m35.086751021s) [noise=true , address=145.79.227.20:4439 , piece_id=W3E226XLYQLNMGY2DSPI67AQNSF62Q6BD3GEOBYLK3VW7BQSIH3Q, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.537679137 (322h26m35.089149599s) [noise=true , address=145.79.227.13:4803 , piece_id=EOOS6U6BBMXIY5C2V2JBRIDF7AA6W2CVGGTFGQWYTE76UHXOCNJQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.539479114 (322h26m35.090949575s) [noise=true , address=145.79.227.21:4473 , piece_id=LNYUOAJ72Y7JBDT46CKO3NJF5VI7BUMFRTHPOVQ37VPSVLMJJWNQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.541172168 (322h26m35.09264263s) [noise=true , address=145.79.227.16:4912 , piece_id=A5T4MWWZWMU7MZCQL5UALJ4NGZC4RRB5KGV65VXEPPX5LRU4PDEQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.541361829 (322h26m35.09283228s) [noise=true , address=145.79.227.9:4366 , piece_id=QUMSUINJ6LPOKUNBJH2SA3O52E74UBQ44VBY5UXGNYYHH56UMWHA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.543412674 (322h26m35.094883125s) [noise=true , address=145.79.227.8:4331 , piece_id=PLOUJIMJXBL3YQ2BFJDSTBZUWAZT65TVOSKEN3LAV6K4T2WOXCYA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.543625580 (322h26m35.095096041s) [noise=true , address=145.79.227.14:4839 , piece_id=U7AVK6SPVJ2YHCQ5JUMXFEYL7YXXOT43GB2SMFC3P3OQ7FJ34CZA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.544819604 (322h26m35.096290055s) [noise=true , address=145.79.227.11:4730 , piece_id=UILM3BBNJBLZXQIAM344ZWKAVNPZP5FOYV7BQFRO3K4X2GY3LT5A, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.545438793 (322h26m35.096909254s) [noise=true , address=77.90.31.21:4879 , piece_id=RC6YSH7QAQSOWU7CZFVBXST6IM625U743W5ARLTNWDAVZEAUS7HQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.546221353 (322h26m35.097691814s) [noise=true , address=145.79.227.32:4836 , piece_id=M42NYA2BQPW3CTOSCWSWCDFJ2YOOFLN3GMJEHIAQLF6FZ2UC3CHA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.547646487 (322h26m35.099116948s) [noise=true , address=45.91.116.5:4298 , piece_id=JIT6FAREK3TLHVJHJBUHBFHPSVFN6WV4XUQKGD3WNLG6O5SZWXNA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.548438445 (322h26m35.099908916s) [noise=true , address=145.79.227.22:4510 , piece_id=KQUMUDY2XK6WO654F6ZUU7QM5IHR2GAPQJYBPB2EBW2OI4MTDVGQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:38.320904919 (322h26m36.8723754s) [upload=283964]: Next returned nil segment Nov 22 18:30:38.320922753 (322h26m36.872393214s) [upload=283964]: Waiting for error group managing segments... Nov 22 18:30:38.649102998 (322h26m37.200573459s) [noise=true , address=45.91.116.4:4940 , piece_id=6LWBQSRMQYU7U3TUGUUMPDIOSX6AWCIO3RE46BXRS73YKR7E25DA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.664944633 (322h26m37.216415114s) [noise=true , address=45.91.116.5:4298 , piece_id=JIT6FAREK3TLHVJHJBUHBFHPSVFN6WV4XUQKGD3WNLG6O5SZWXNA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.667959895 (322h26m37.219430356s) [noise=true , address=77.90.31.13:4627 , piece_id=VJFT5OTJEZQYSPRL4QHAWFWF7HCRYMUZDESZO4BJM4NW5XZL7ZQQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.768748473 (322h26m37.320218954s) [noise=true , address=77.90.31.21:4879 , piece_id=RC6YSH7QAQSOWU7CZFVBXST6IM625U743W5ARLTNWDAVZEAUS7HQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:40.165076524 (322h26m38.716546975s) [noise=true , address=145.79.227.22:4510 , piece_id=KQUMUDY2XK6WO654F6ZUU7QM5IHR2GAPQJYBPB2EBW2OI4MTDVGQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:40.965617167 (322h26m39.517087629s) [noise=true , address=145.79.227.14:4839 , piece_id=U7AVK6SPVJ2YHCQ5JUMXFEYL7YXXOT43GB2SMFC3P3OQ7FJ34CZA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:42.701227963 (322h26m41.252698424s) [noise=true , address=145.79.227.9:4366 , piece_id=QUMSUINJ6LPOKUNBJH2SA3O52E74UBQ44VBY5UXGNYYHH56UMWHA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:43.663026451 (322h26m42.214496902s) [noise=true , address=145.79.227.13:4803 , piece_id=EOOS6U6BBMXIY5C2V2JBRIDF7AA6W2CVGGTFGQWYTE76UHXOCNJQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:45.482181600 (322h26m44.033652061s) [noise=true , address=145.79.227.20:4439 , piece_id=W3E226XLYQLNMGY2DSPI67AQNSF62Q6BD3GEOBYLK3VW7BQSIH3Q, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:48.700608997 (322h26m47.252079458s) [noise=true , address=145.79.227.21:4473 , piece_id=LNYUOAJ72Y7JBDT46CKO3NJF5VI7BUMFRTHPOVQ37VPSVLMJJWNQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:31:05.586366500 (322h27m4.137836971s) [noise=true , address=145.79.227.11:4730 , piece_id=UILM3BBNJBLZXQIAM344ZWKAVNPZP5FOYV7BQFRO3K4X2GY3LT5A, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection Nov 22 18:31:05.586387008 (322h27m4.137857469s) [noise=true , address=145.79.227.16:4912 , piece_id=A5T4MWWZWMU7MZCQL5UALJ4NGZC4RRB5KGV65VXEPPX5LRU4PDEQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection Nov 22 18:31:05.586537225 (322h27m4.138007686s) [noise=true , address=145.79.227.32:4836 , piece_id=M42NYA2BQPW3CTOSCWSWCDFJ2YOOFLN3GMJEHIAQLF6FZ2UC3CHA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection Nov 22 18:31:05.586630993 (322h27m4.138101454s) [noise=true , address=145.79.227.17:4970 , piece_id=7JXIJJROS4CCTDQOTOOL6XCNBA3ASBVGCO4YF75G5QRJ2QY2QYAA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection Nov 22 18:31:05.586783464 (322h27m4.138253915s) [noise=true , address=145.79.227.15:4875 , piece_id=HNFPLSNQQQM5M5I7J44WCAQL2XLRQ4GM44IHI7XI2ESNDCPFDWIQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection Nov 22 18:31:05.587158868 (322h27m4.138629349s) [noise=true , address=145.79.227.8:4331 , piece_id=PLOUJIMJXBL3YQ2BFJDSTBZUWAZT65TVOSKEN3LAV6K4T2WOXCYA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:37548->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 18:31:05.587230965 (322h27m4.138701426s) [seg_pos={0 0}, upload=283964]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 18:31:05.833640584 (322h27m4.385111055s) [upload=283964]: Done uploading segments... Nov 22 18:31:05.833691370 (322h27m4.385161841s) [upload=283964]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 23.943695327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.993015988s trace: https://jaeger.dev.storj.io/trace/5225b8d0a62143ca FAIL (2025-11-22 18:35:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== s) [upload=283964]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:30:36.531047319 (322h26m35.08251778s) [seg_pos={0 0}, upload=283964]: Begin upload segment... Nov 22 18:30:36.531092805 (322h26m35.082563256s) [seg_pos={0 0}, upload=283964]: Done begin upload segment. Nov 22 18:30:36.531122301 (322h26m35.082592763s) [noise=true , address=77.90.31.13:4627 , piece_id=VJFT5OTJEZQYSPRL4QHAWFWF7HCRYMUZDESZO4BJM4NW5XZL7ZQQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.532650812 (322h26m35.084121263s) [noise=true , address=145.79.227.17:4970 , piece_id=7JXIJJROS4CCTDQOTOOL6XCNBA3ASBVGCO4YF75G5QRJ2QY2QYAA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.533640798 (322h26m35.085111259s) [noise=true , address=145.79.227.15:4875 , piece_id=HNFPLSNQQQM5M5I7J44WCAQL2XLRQ4GM44IHI7XI2ESNDCPFDWIQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.534514271 (322h26m35.085984732s) [noise=true , address=45.91.116.4:4940 , piece_id=6LWBQSRMQYU7U3TUGUUMPDIOSX6AWCIO3RE46BXRS73YKR7E25DA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.535280560 (322h26m35.086751021s) [noise=true , address=145.79.227.20:4439 , piece_id=W3E226XLYQLNMGY2DSPI67AQNSF62Q6BD3GEOBYLK3VW7BQSIH3Q, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.537679137 (322h26m35.089149599s) [noise=true , address=145.79.227.13:4803 , piece_id=EOOS6U6BBMXIY5C2V2JBRIDF7AA6W2CVGGTFGQWYTE76UHXOCNJQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.539479114 (322h26m35.090949575s) [noise=true , address=145.79.227.21:4473 , piece_id=LNYUOAJ72Y7JBDT46CKO3NJF5VI7BUMFRTHPOVQ37VPSVLMJJWNQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.541172168 (322h26m35.09264263s) [noise=true , address=145.79.227.16:4912 , piece_id=A5T4MWWZWMU7MZCQL5UALJ4NGZC4RRB5KGV65VXEPPX5LRU4PDEQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.541361829 (322h26m35.09283228s) [noise=true , address=145.79.227.9:4366 , piece_id=QUMSUINJ6LPOKUNBJH2SA3O52E74UBQ44VBY5UXGNYYHH56UMWHA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.543412674 (322h26m35.094883125s) [noise=true , address=145.79.227.8:4331 , piece_id=PLOUJIMJXBL3YQ2BFJDSTBZUWAZT65TVOSKEN3LAV6K4T2WOXCYA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.543625580 (322h26m35.095096041s) [noise=true , address=145.79.227.14:4839 , piece_id=U7AVK6SPVJ2YHCQ5JUMXFEYL7YXXOT43GB2SMFC3P3OQ7FJ34CZA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.544819604 (322h26m35.096290055s) [noise=true , address=145.79.227.11:4730 , piece_id=UILM3BBNJBLZXQIAM344ZWKAVNPZP5FOYV7BQFRO3K4X2GY3LT5A, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.545438793 (322h26m35.096909254s) [noise=true , address=77.90.31.21:4879 , piece_id=RC6YSH7QAQSOWU7CZFVBXST6IM625U743W5ARLTNWDAVZEAUS7HQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.546221353 (322h26m35.097691814s) [noise=true , address=145.79.227.32:4836 , piece_id=M42NYA2BQPW3CTOSCWSWCDFJ2YOOFLN3GMJEHIAQLF6FZ2UC3CHA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.547646487 (322h26m35.099116948s) [noise=true , address=45.91.116.5:4298 , piece_id=JIT6FAREK3TLHVJHJBUHBFHPSVFN6WV4XUQKGD3WNLG6O5SZWXNA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.548438445 (322h26m35.099908916s) [noise=true , address=145.79.227.22:4510 , piece_id=KQUMUDY2XK6WO654F6ZUU7QM5IHR2GAPQJYBPB2EBW2OI4MTDVGQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:38.320904919 (322h26m36.8723754s) [upload=283964]: Next returned nil segment Nov 22 18:30:38.320922753 (322h26m36.872393214s) [upload=283964]: Waiting for error group managing segments... Nov 22 18:30:38.649102998 (322h26m37.200573459s) [noise=true , address=45.91.116.4:4940 , piece_id=6LWBQSRMQYU7U3TUGUUMPDIOSX6AWCIO3RE46BXRS73YKR7E25DA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.664944633 (322h26m37.216415114s) [noise=true , address=45.91.116.5:4298 , piece_id=JIT6FAREK3TLHVJHJBUHBFHPSVFN6WV4XUQKGD3WNLG6O5SZWXNA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.667959895 (322h26m37.219430356s) [noise=true , address=77.90.31.13:4627 , piece_id=VJFT5OTJEZQYSPRL4QHAWFWF7HCRYMUZDESZO4BJM4NW5XZL7ZQQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.768748473 (322h26m37.320218954s) [noise=true , address=77.90.31.21:4879 , piece_id=RC6YSH7QAQSOWU7CZFVBXST6IM625U743W5ARLTNWDAVZEAUS7HQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:40.165076524 (322h26m38.716546975s) [noise=true , address=145.79.227.22:4510 , piece_id=KQUMUDY2XK6WO654F6ZUU7QM5IHR2GAPQJYBPB2EBW2OI4MTDVGQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:40.965617167 (322h26m39.517087629s) [noise=true , address=145.79.227.14:4839 , piece_id=U7AVK6SPVJ2YHCQ5JUMXFEYL7YXXOT43GB2SMFC3P3OQ7FJ34CZA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:42.701227963 (322h26m41.252698424s) [noise=true , address=145.79.227.9:4366 , piece_id=QUMSUINJ6LPOKUNBJH2SA3O52E74UBQ44VBY5UXGNYYHH56UMWHA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:43.663026451 (322h26m42.214496902s) [noise=true , address=145.79.227.13:4803 , piece_id=EOOS6U6BBMXIY5C2V2JBRIDF7AA6W2CVGGTFGQWYTE76UHXOCNJQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:45.482181600 (322h26m44.033652061s) [noise=true , address=145.79.227.20:4439 , piece_id=W3E226XLYQLNMGY2DSPI67AQNSF62Q6BD3GEOBYLK3VW7BQSIH3Q, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:48.700608997 (322h26m47.252079458s) [noise=true , address=145.79.227.21:4473 , piece_id=LNYUOAJ72Y7JBDT46CKO3NJF5VI7BUMFRTHPOVQ37VPSVLMJJWNQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:31:05.586366500 (322h27m4.137836971s) [noise=true , address=145.79.227.11:4730 , piece_id=UILM3BBNJBLZXQIAM344ZWKAVNPZP5FOYV7BQFRO3K4X2GY3LT5A, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection Nov 22 18:31:05.586387008 (322h27m4.137857469s) [noise=true , address=145.79.227.16:4912 , piece_id=A5T4MWWZWMU7MZCQL5UALJ4NGZC4RRB5KGV65VXEPPX5LRU4PDEQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection Nov 22 18:31:05.586537225 (322h27m4.138007686s) [noise=true , address=145.79.227.32:4836 , piece_id=M42NYA2BQPW3CTOSCWSWCDFJ2YOOFLN3GMJEHIAQLF6FZ2UC3CHA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection Nov 22 18:31:05.586630993 (322h27m4.138101454s) [noise=true , address=145.79.227.17:4970 , piece_id=7JXIJJROS4CCTDQOTOOL6XCNBA3ASBVGCO4YF75G5QRJ2QY2QYAA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection Nov 22 18:31:05.586783464 (322h27m4.138253915s) [noise=true , address=145.79.227.15:4875 , piece_id=HNFPLSNQQQM5M5I7J44WCAQL2XLRQ4GM44IHI7XI2ESNDCPFDWIQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection Nov 22 18:31:05.587158868 (322h27m4.138629349s) [noise=true , address=145.79.227.8:4331 , piece_id=PLOUJIMJXBL3YQ2BFJDSTBZUWAZT65TVOSKEN3LAV6K4T2WOXCYA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:37548->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 18:31:05.587230965 (322h27m4.138701426s) [seg_pos={0 0}, upload=283964]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 18:31:05.833640584 (322h27m4.385111055s) [upload=283964]: Done uploading segments... Nov 22 18:31:05.833691370 (322h27m4.385161841s) [upload=283964]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 5.993041377s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.856323306s trace: https://jaeger.dev.storj.io/trace/5225b8d0a62143ca FAIL (2025-11-22 18:34:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== s) [upload=283964]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:30:36.531047319 (322h26m35.08251778s) [seg_pos={0 0}, upload=283964]: Begin upload segment... Nov 22 18:30:36.531092805 (322h26m35.082563256s) [seg_pos={0 0}, upload=283964]: Done begin upload segment. Nov 22 18:30:36.531122301 (322h26m35.082592763s) [noise=true , address=77.90.31.13:4627 , piece_id=VJFT5OTJEZQYSPRL4QHAWFWF7HCRYMUZDESZO4BJM4NW5XZL7ZQQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.532650812 (322h26m35.084121263s) [noise=true , address=145.79.227.17:4970 , piece_id=7JXIJJROS4CCTDQOTOOL6XCNBA3ASBVGCO4YF75G5QRJ2QY2QYAA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.533640798 (322h26m35.085111259s) [noise=true , address=145.79.227.15:4875 , piece_id=HNFPLSNQQQM5M5I7J44WCAQL2XLRQ4GM44IHI7XI2ESNDCPFDWIQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.534514271 (322h26m35.085984732s) [noise=true , address=45.91.116.4:4940 , piece_id=6LWBQSRMQYU7U3TUGUUMPDIOSX6AWCIO3RE46BXRS73YKR7E25DA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.535280560 (322h26m35.086751021s) [noise=true , address=145.79.227.20:4439 , piece_id=W3E226XLYQLNMGY2DSPI67AQNSF62Q6BD3GEOBYLK3VW7BQSIH3Q, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.537679137 (322h26m35.089149599s) [noise=true , address=145.79.227.13:4803 , piece_id=EOOS6U6BBMXIY5C2V2JBRIDF7AA6W2CVGGTFGQWYTE76UHXOCNJQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.539479114 (322h26m35.090949575s) [noise=true , address=145.79.227.21:4473 , piece_id=LNYUOAJ72Y7JBDT46CKO3NJF5VI7BUMFRTHPOVQ37VPSVLMJJWNQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.541172168 (322h26m35.09264263s) [noise=true , address=145.79.227.16:4912 , piece_id=A5T4MWWZWMU7MZCQL5UALJ4NGZC4RRB5KGV65VXEPPX5LRU4PDEQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.541361829 (322h26m35.09283228s) [noise=true , address=145.79.227.9:4366 , piece_id=QUMSUINJ6LPOKUNBJH2SA3O52E74UBQ44VBY5UXGNYYHH56UMWHA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.543412674 (322h26m35.094883125s) [noise=true , address=145.79.227.8:4331 , piece_id=PLOUJIMJXBL3YQ2BFJDSTBZUWAZT65TVOSKEN3LAV6K4T2WOXCYA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.543625580 (322h26m35.095096041s) [noise=true , address=145.79.227.14:4839 , piece_id=U7AVK6SPVJ2YHCQ5JUMXFEYL7YXXOT43GB2SMFC3P3OQ7FJ34CZA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.544819604 (322h26m35.096290055s) [noise=true , address=145.79.227.11:4730 , piece_id=UILM3BBNJBLZXQIAM344ZWKAVNPZP5FOYV7BQFRO3K4X2GY3LT5A, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.545438793 (322h26m35.096909254s) [noise=true , address=77.90.31.21:4879 , piece_id=RC6YSH7QAQSOWU7CZFVBXST6IM625U743W5ARLTNWDAVZEAUS7HQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.546221353 (322h26m35.097691814s) [noise=true , address=145.79.227.32:4836 , piece_id=M42NYA2BQPW3CTOSCWSWCDFJ2YOOFLN3GMJEHIAQLF6FZ2UC3CHA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.547646487 (322h26m35.099116948s) [noise=true , address=45.91.116.5:4298 , piece_id=JIT6FAREK3TLHVJHJBUHBFHPSVFN6WV4XUQKGD3WNLG6O5SZWXNA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.548438445 (322h26m35.099908916s) [noise=true , address=145.79.227.22:4510 , piece_id=KQUMUDY2XK6WO654F6ZUU7QM5IHR2GAPQJYBPB2EBW2OI4MTDVGQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:38.320904919 (322h26m36.8723754s) [upload=283964]: Next returned nil segment Nov 22 18:30:38.320922753 (322h26m36.872393214s) [upload=283964]: Waiting for error group managing segments... Nov 22 18:30:38.649102998 (322h26m37.200573459s) [noise=true , address=45.91.116.4:4940 , piece_id=6LWBQSRMQYU7U3TUGUUMPDIOSX6AWCIO3RE46BXRS73YKR7E25DA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.664944633 (322h26m37.216415114s) [noise=true , address=45.91.116.5:4298 , piece_id=JIT6FAREK3TLHVJHJBUHBFHPSVFN6WV4XUQKGD3WNLG6O5SZWXNA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.667959895 (322h26m37.219430356s) [noise=true , address=77.90.31.13:4627 , piece_id=VJFT5OTJEZQYSPRL4QHAWFWF7HCRYMUZDESZO4BJM4NW5XZL7ZQQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.768748473 (322h26m37.320218954s) [noise=true , address=77.90.31.21:4879 , piece_id=RC6YSH7QAQSOWU7CZFVBXST6IM625U743W5ARLTNWDAVZEAUS7HQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:40.165076524 (322h26m38.716546975s) [noise=true , address=145.79.227.22:4510 , piece_id=KQUMUDY2XK6WO654F6ZUU7QM5IHR2GAPQJYBPB2EBW2OI4MTDVGQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:40.965617167 (322h26m39.517087629s) [noise=true , address=145.79.227.14:4839 , piece_id=U7AVK6SPVJ2YHCQ5JUMXFEYL7YXXOT43GB2SMFC3P3OQ7FJ34CZA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:42.701227963 (322h26m41.252698424s) [noise=true , address=145.79.227.9:4366 , piece_id=QUMSUINJ6LPOKUNBJH2SA3O52E74UBQ44VBY5UXGNYYHH56UMWHA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:43.663026451 (322h26m42.214496902s) [noise=true , address=145.79.227.13:4803 , piece_id=EOOS6U6BBMXIY5C2V2JBRIDF7AA6W2CVGGTFGQWYTE76UHXOCNJQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:45.482181600 (322h26m44.033652061s) [noise=true , address=145.79.227.20:4439 , piece_id=W3E226XLYQLNMGY2DSPI67AQNSF62Q6BD3GEOBYLK3VW7BQSIH3Q, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:48.700608997 (322h26m47.252079458s) [noise=true , address=145.79.227.21:4473 , piece_id=LNYUOAJ72Y7JBDT46CKO3NJF5VI7BUMFRTHPOVQ37VPSVLMJJWNQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:31:05.586366500 (322h27m4.137836971s) [noise=true , address=145.79.227.11:4730 , piece_id=UILM3BBNJBLZXQIAM344ZWKAVNPZP5FOYV7BQFRO3K4X2GY3LT5A, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection Nov 22 18:31:05.586387008 (322h27m4.137857469s) [noise=true , address=145.79.227.16:4912 , piece_id=A5T4MWWZWMU7MZCQL5UALJ4NGZC4RRB5KGV65VXEPPX5LRU4PDEQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection Nov 22 18:31:05.586537225 (322h27m4.138007686s) [noise=true , address=145.79.227.32:4836 , piece_id=M42NYA2BQPW3CTOSCWSWCDFJ2YOOFLN3GMJEHIAQLF6FZ2UC3CHA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection Nov 22 18:31:05.586630993 (322h27m4.138101454s) [noise=true , address=145.79.227.17:4970 , piece_id=7JXIJJROS4CCTDQOTOOL6XCNBA3ASBVGCO4YF75G5QRJ2QY2QYAA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection Nov 22 18:31:05.586783464 (322h27m4.138253915s) [noise=true , address=145.79.227.15:4875 , piece_id=HNFPLSNQQQM5M5I7J44WCAQL2XLRQ4GM44IHI7XI2ESNDCPFDWIQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection Nov 22 18:31:05.587158868 (322h27m4.138629349s) [noise=true , address=145.79.227.8:4331 , piece_id=PLOUJIMJXBL3YQ2BFJDSTBZUWAZT65TVOSKEN3LAV6K4T2WOXCYA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:37548->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 18:31:05.587230965 (322h27m4.138701426s) [seg_pos={0 0}, upload=283964]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 18:31:05.833640584 (322h27m4.385111055s) [upload=283964]: Done uploading segments... Nov 22 18:31:05.833691370 (322h27m4.385161841s) [upload=283964]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 5.856355106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.098326392s trace: https://jaeger.dev.storj.io/trace/5225b8d0a62143ca FAIL (2025-11-22 18:33:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== s) [upload=283964]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:30:36.531047319 (322h26m35.08251778s) [seg_pos={0 0}, upload=283964]: Begin upload segment... Nov 22 18:30:36.531092805 (322h26m35.082563256s) [seg_pos={0 0}, upload=283964]: Done begin upload segment. Nov 22 18:30:36.531122301 (322h26m35.082592763s) [noise=true , address=77.90.31.13:4627 , piece_id=VJFT5OTJEZQYSPRL4QHAWFWF7HCRYMUZDESZO4BJM4NW5XZL7ZQQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.532650812 (322h26m35.084121263s) [noise=true , address=145.79.227.17:4970 , piece_id=7JXIJJROS4CCTDQOTOOL6XCNBA3ASBVGCO4YF75G5QRJ2QY2QYAA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.533640798 (322h26m35.085111259s) [noise=true , address=145.79.227.15:4875 , piece_id=HNFPLSNQQQM5M5I7J44WCAQL2XLRQ4GM44IHI7XI2ESNDCPFDWIQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.534514271 (322h26m35.085984732s) [noise=true , address=45.91.116.4:4940 , piece_id=6LWBQSRMQYU7U3TUGUUMPDIOSX6AWCIO3RE46BXRS73YKR7E25DA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.535280560 (322h26m35.086751021s) [noise=true , address=145.79.227.20:4439 , piece_id=W3E226XLYQLNMGY2DSPI67AQNSF62Q6BD3GEOBYLK3VW7BQSIH3Q, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.537679137 (322h26m35.089149599s) [noise=true , address=145.79.227.13:4803 , piece_id=EOOS6U6BBMXIY5C2V2JBRIDF7AA6W2CVGGTFGQWYTE76UHXOCNJQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.539479114 (322h26m35.090949575s) [noise=true , address=145.79.227.21:4473 , piece_id=LNYUOAJ72Y7JBDT46CKO3NJF5VI7BUMFRTHPOVQ37VPSVLMJJWNQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.541172168 (322h26m35.09264263s) [noise=true , address=145.79.227.16:4912 , piece_id=A5T4MWWZWMU7MZCQL5UALJ4NGZC4RRB5KGV65VXEPPX5LRU4PDEQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.541361829 (322h26m35.09283228s) [noise=true , address=145.79.227.9:4366 , piece_id=QUMSUINJ6LPOKUNBJH2SA3O52E74UBQ44VBY5UXGNYYHH56UMWHA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.543412674 (322h26m35.094883125s) [noise=true , address=145.79.227.8:4331 , piece_id=PLOUJIMJXBL3YQ2BFJDSTBZUWAZT65TVOSKEN3LAV6K4T2WOXCYA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.543625580 (322h26m35.095096041s) [noise=true , address=145.79.227.14:4839 , piece_id=U7AVK6SPVJ2YHCQ5JUMXFEYL7YXXOT43GB2SMFC3P3OQ7FJ34CZA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.544819604 (322h26m35.096290055s) [noise=true , address=145.79.227.11:4730 , piece_id=UILM3BBNJBLZXQIAM344ZWKAVNPZP5FOYV7BQFRO3K4X2GY3LT5A, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.545438793 (322h26m35.096909254s) [noise=true , address=77.90.31.21:4879 , piece_id=RC6YSH7QAQSOWU7CZFVBXST6IM625U743W5ARLTNWDAVZEAUS7HQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.546221353 (322h26m35.097691814s) [noise=true , address=145.79.227.32:4836 , piece_id=M42NYA2BQPW3CTOSCWSWCDFJ2YOOFLN3GMJEHIAQLF6FZ2UC3CHA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.547646487 (322h26m35.099116948s) [noise=true , address=45.91.116.5:4298 , piece_id=JIT6FAREK3TLHVJHJBUHBFHPSVFN6WV4XUQKGD3WNLG6O5SZWXNA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.548438445 (322h26m35.099908916s) [noise=true , address=145.79.227.22:4510 , piece_id=KQUMUDY2XK6WO654F6ZUU7QM5IHR2GAPQJYBPB2EBW2OI4MTDVGQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:38.320904919 (322h26m36.8723754s) [upload=283964]: Next returned nil segment Nov 22 18:30:38.320922753 (322h26m36.872393214s) [upload=283964]: Waiting for error group managing segments... Nov 22 18:30:38.649102998 (322h26m37.200573459s) [noise=true , address=45.91.116.4:4940 , piece_id=6LWBQSRMQYU7U3TUGUUMPDIOSX6AWCIO3RE46BXRS73YKR7E25DA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.664944633 (322h26m37.216415114s) [noise=true , address=45.91.116.5:4298 , piece_id=JIT6FAREK3TLHVJHJBUHBFHPSVFN6WV4XUQKGD3WNLG6O5SZWXNA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.667959895 (322h26m37.219430356s) [noise=true , address=77.90.31.13:4627 , piece_id=VJFT5OTJEZQYSPRL4QHAWFWF7HCRYMUZDESZO4BJM4NW5XZL7ZQQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.768748473 (322h26m37.320218954s) [noise=true , address=77.90.31.21:4879 , piece_id=RC6YSH7QAQSOWU7CZFVBXST6IM625U743W5ARLTNWDAVZEAUS7HQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:40.165076524 (322h26m38.716546975s) [noise=true , address=145.79.227.22:4510 , piece_id=KQUMUDY2XK6WO654F6ZUU7QM5IHR2GAPQJYBPB2EBW2OI4MTDVGQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:40.965617167 (322h26m39.517087629s) [noise=true , address=145.79.227.14:4839 , piece_id=U7AVK6SPVJ2YHCQ5JUMXFEYL7YXXOT43GB2SMFC3P3OQ7FJ34CZA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:42.701227963 (322h26m41.252698424s) [noise=true , address=145.79.227.9:4366 , piece_id=QUMSUINJ6LPOKUNBJH2SA3O52E74UBQ44VBY5UXGNYYHH56UMWHA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:43.663026451 (322h26m42.214496902s) [noise=true , address=145.79.227.13:4803 , piece_id=EOOS6U6BBMXIY5C2V2JBRIDF7AA6W2CVGGTFGQWYTE76UHXOCNJQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:45.482181600 (322h26m44.033652061s) [noise=true , address=145.79.227.20:4439 , piece_id=W3E226XLYQLNMGY2DSPI67AQNSF62Q6BD3GEOBYLK3VW7BQSIH3Q, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:48.700608997 (322h26m47.252079458s) [noise=true , address=145.79.227.21:4473 , piece_id=LNYUOAJ72Y7JBDT46CKO3NJF5VI7BUMFRTHPOVQ37VPSVLMJJWNQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:31:05.586366500 (322h27m4.137836971s) [noise=true , address=145.79.227.11:4730 , piece_id=UILM3BBNJBLZXQIAM344ZWKAVNPZP5FOYV7BQFRO3K4X2GY3LT5A, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection Nov 22 18:31:05.586387008 (322h27m4.137857469s) [noise=true , address=145.79.227.16:4912 , piece_id=A5T4MWWZWMU7MZCQL5UALJ4NGZC4RRB5KGV65VXEPPX5LRU4PDEQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection Nov 22 18:31:05.586537225 (322h27m4.138007686s) [noise=true , address=145.79.227.32:4836 , piece_id=M42NYA2BQPW3CTOSCWSWCDFJ2YOOFLN3GMJEHIAQLF6FZ2UC3CHA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection Nov 22 18:31:05.586630993 (322h27m4.138101454s) [noise=true , address=145.79.227.17:4970 , piece_id=7JXIJJROS4CCTDQOTOOL6XCNBA3ASBVGCO4YF75G5QRJ2QY2QYAA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection Nov 22 18:31:05.586783464 (322h27m4.138253915s) [noise=true , address=145.79.227.15:4875 , piece_id=HNFPLSNQQQM5M5I7J44WCAQL2XLRQ4GM44IHI7XI2ESNDCPFDWIQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection Nov 22 18:31:05.587158868 (322h27m4.138629349s) [noise=true , address=145.79.227.8:4331 , piece_id=PLOUJIMJXBL3YQ2BFJDSTBZUWAZT65TVOSKEN3LAV6K4T2WOXCYA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:37548->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 18:31:05.587230965 (322h27m4.138701426s) [seg_pos={0 0}, upload=283964]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 18:31:05.833640584 (322h27m4.385111055s) [upload=283964]: Done uploading segments... Nov 22 18:31:05.833691370 (322h27m4.385161841s) [upload=283964]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 6.098359395s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.687710947s trace: https://jaeger.dev.storj.io/trace/5225b8d0a62143ca FAIL (2025-11-22 18:32:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== s) [upload=283964]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:30:36.531047319 (322h26m35.08251778s) [seg_pos={0 0}, upload=283964]: Begin upload segment... Nov 22 18:30:36.531092805 (322h26m35.082563256s) [seg_pos={0 0}, upload=283964]: Done begin upload segment. Nov 22 18:30:36.531122301 (322h26m35.082592763s) [noise=true , address=77.90.31.13:4627 , piece_id=VJFT5OTJEZQYSPRL4QHAWFWF7HCRYMUZDESZO4BJM4NW5XZL7ZQQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.532650812 (322h26m35.084121263s) [noise=true , address=145.79.227.17:4970 , piece_id=7JXIJJROS4CCTDQOTOOL6XCNBA3ASBVGCO4YF75G5QRJ2QY2QYAA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.533640798 (322h26m35.085111259s) [noise=true , address=145.79.227.15:4875 , piece_id=HNFPLSNQQQM5M5I7J44WCAQL2XLRQ4GM44IHI7XI2ESNDCPFDWIQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.534514271 (322h26m35.085984732s) [noise=true , address=45.91.116.4:4940 , piece_id=6LWBQSRMQYU7U3TUGUUMPDIOSX6AWCIO3RE46BXRS73YKR7E25DA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.535280560 (322h26m35.086751021s) [noise=true , address=145.79.227.20:4439 , piece_id=W3E226XLYQLNMGY2DSPI67AQNSF62Q6BD3GEOBYLK3VW7BQSIH3Q, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.537679137 (322h26m35.089149599s) [noise=true , address=145.79.227.13:4803 , piece_id=EOOS6U6BBMXIY5C2V2JBRIDF7AA6W2CVGGTFGQWYTE76UHXOCNJQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.539479114 (322h26m35.090949575s) [noise=true , address=145.79.227.21:4473 , piece_id=LNYUOAJ72Y7JBDT46CKO3NJF5VI7BUMFRTHPOVQ37VPSVLMJJWNQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.541172168 (322h26m35.09264263s) [noise=true , address=145.79.227.16:4912 , piece_id=A5T4MWWZWMU7MZCQL5UALJ4NGZC4RRB5KGV65VXEPPX5LRU4PDEQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.541361829 (322h26m35.09283228s) [noise=true , address=145.79.227.9:4366 , piece_id=QUMSUINJ6LPOKUNBJH2SA3O52E74UBQ44VBY5UXGNYYHH56UMWHA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.543412674 (322h26m35.094883125s) [noise=true , address=145.79.227.8:4331 , piece_id=PLOUJIMJXBL3YQ2BFJDSTBZUWAZT65TVOSKEN3LAV6K4T2WOXCYA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.543625580 (322h26m35.095096041s) [noise=true , address=145.79.227.14:4839 , piece_id=U7AVK6SPVJ2YHCQ5JUMXFEYL7YXXOT43GB2SMFC3P3OQ7FJ34CZA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.544819604 (322h26m35.096290055s) [noise=true , address=145.79.227.11:4730 , piece_id=UILM3BBNJBLZXQIAM344ZWKAVNPZP5FOYV7BQFRO3K4X2GY3LT5A, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.545438793 (322h26m35.096909254s) [noise=true , address=77.90.31.21:4879 , piece_id=RC6YSH7QAQSOWU7CZFVBXST6IM625U743W5ARLTNWDAVZEAUS7HQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.546221353 (322h26m35.097691814s) [noise=true , address=145.79.227.32:4836 , piece_id=M42NYA2BQPW3CTOSCWSWCDFJ2YOOFLN3GMJEHIAQLF6FZ2UC3CHA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.547646487 (322h26m35.099116948s) [noise=true , address=45.91.116.5:4298 , piece_id=JIT6FAREK3TLHVJHJBUHBFHPSVFN6WV4XUQKGD3WNLG6O5SZWXNA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.548438445 (322h26m35.099908916s) [noise=true , address=145.79.227.22:4510 , piece_id=KQUMUDY2XK6WO654F6ZUU7QM5IHR2GAPQJYBPB2EBW2OI4MTDVGQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:38.320904919 (322h26m36.8723754s) [upload=283964]: Next returned nil segment Nov 22 18:30:38.320922753 (322h26m36.872393214s) [upload=283964]: Waiting for error group managing segments... Nov 22 18:30:38.649102998 (322h26m37.200573459s) [noise=true , address=45.91.116.4:4940 , piece_id=6LWBQSRMQYU7U3TUGUUMPDIOSX6AWCIO3RE46BXRS73YKR7E25DA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.664944633 (322h26m37.216415114s) [noise=true , address=45.91.116.5:4298 , piece_id=JIT6FAREK3TLHVJHJBUHBFHPSVFN6WV4XUQKGD3WNLG6O5SZWXNA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.667959895 (322h26m37.219430356s) [noise=true , address=77.90.31.13:4627 , piece_id=VJFT5OTJEZQYSPRL4QHAWFWF7HCRYMUZDESZO4BJM4NW5XZL7ZQQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.768748473 (322h26m37.320218954s) [noise=true , address=77.90.31.21:4879 , piece_id=RC6YSH7QAQSOWU7CZFVBXST6IM625U743W5ARLTNWDAVZEAUS7HQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:40.165076524 (322h26m38.716546975s) [noise=true , address=145.79.227.22:4510 , piece_id=KQUMUDY2XK6WO654F6ZUU7QM5IHR2GAPQJYBPB2EBW2OI4MTDVGQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:40.965617167 (322h26m39.517087629s) [noise=true , address=145.79.227.14:4839 , piece_id=U7AVK6SPVJ2YHCQ5JUMXFEYL7YXXOT43GB2SMFC3P3OQ7FJ34CZA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:42.701227963 (322h26m41.252698424s) [noise=true , address=145.79.227.9:4366 , piece_id=QUMSUINJ6LPOKUNBJH2SA3O52E74UBQ44VBY5UXGNYYHH56UMWHA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:43.663026451 (322h26m42.214496902s) [noise=true , address=145.79.227.13:4803 , piece_id=EOOS6U6BBMXIY5C2V2JBRIDF7AA6W2CVGGTFGQWYTE76UHXOCNJQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:45.482181600 (322h26m44.033652061s) [noise=true , address=145.79.227.20:4439 , piece_id=W3E226XLYQLNMGY2DSPI67AQNSF62Q6BD3GEOBYLK3VW7BQSIH3Q, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:48.700608997 (322h26m47.252079458s) [noise=true , address=145.79.227.21:4473 , piece_id=LNYUOAJ72Y7JBDT46CKO3NJF5VI7BUMFRTHPOVQ37VPSVLMJJWNQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:31:05.586366500 (322h27m4.137836971s) [noise=true , address=145.79.227.11:4730 , piece_id=UILM3BBNJBLZXQIAM344ZWKAVNPZP5FOYV7BQFRO3K4X2GY3LT5A, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection Nov 22 18:31:05.586387008 (322h27m4.137857469s) [noise=true , address=145.79.227.16:4912 , piece_id=A5T4MWWZWMU7MZCQL5UALJ4NGZC4RRB5KGV65VXEPPX5LRU4PDEQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection Nov 22 18:31:05.586537225 (322h27m4.138007686s) [noise=true , address=145.79.227.32:4836 , piece_id=M42NYA2BQPW3CTOSCWSWCDFJ2YOOFLN3GMJEHIAQLF6FZ2UC3CHA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection Nov 22 18:31:05.586630993 (322h27m4.138101454s) [noise=true , address=145.79.227.17:4970 , piece_id=7JXIJJROS4CCTDQOTOOL6XCNBA3ASBVGCO4YF75G5QRJ2QY2QYAA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection Nov 22 18:31:05.586783464 (322h27m4.138253915s) [noise=true , address=145.79.227.15:4875 , piece_id=HNFPLSNQQQM5M5I7J44WCAQL2XLRQ4GM44IHI7XI2ESNDCPFDWIQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection Nov 22 18:31:05.587158868 (322h27m4.138629349s) [noise=true , address=145.79.227.8:4331 , piece_id=PLOUJIMJXBL3YQ2BFJDSTBZUWAZT65TVOSKEN3LAV6K4T2WOXCYA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:37548->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 18:31:05.587230965 (322h27m4.138701426s) [seg_pos={0 0}, upload=283964]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 18:31:05.833640584 (322h27m4.385111055s) [upload=283964]: Done uploading segments... Nov 22 18:31:05.833691370 (322h27m4.385161841s) [upload=283964]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 5.687749008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.838943232s trace: https://jaeger.dev.storj.io/trace/5225b8d0a62143ca FAIL (2025-11-22 18:31:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== s) [upload=283964]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:30:36.531047319 (322h26m35.08251778s) [seg_pos={0 0}, upload=283964]: Begin upload segment... Nov 22 18:30:36.531092805 (322h26m35.082563256s) [seg_pos={0 0}, upload=283964]: Done begin upload segment. Nov 22 18:30:36.531122301 (322h26m35.082592763s) [noise=true , address=77.90.31.13:4627 , piece_id=VJFT5OTJEZQYSPRL4QHAWFWF7HCRYMUZDESZO4BJM4NW5XZL7ZQQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.532650812 (322h26m35.084121263s) [noise=true , address=145.79.227.17:4970 , piece_id=7JXIJJROS4CCTDQOTOOL6XCNBA3ASBVGCO4YF75G5QRJ2QY2QYAA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.533640798 (322h26m35.085111259s) [noise=true , address=145.79.227.15:4875 , piece_id=HNFPLSNQQQM5M5I7J44WCAQL2XLRQ4GM44IHI7XI2ESNDCPFDWIQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.534514271 (322h26m35.085984732s) [noise=true , address=45.91.116.4:4940 , piece_id=6LWBQSRMQYU7U3TUGUUMPDIOSX6AWCIO3RE46BXRS73YKR7E25DA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.535280560 (322h26m35.086751021s) [noise=true , address=145.79.227.20:4439 , piece_id=W3E226XLYQLNMGY2DSPI67AQNSF62Q6BD3GEOBYLK3VW7BQSIH3Q, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.537679137 (322h26m35.089149599s) [noise=true , address=145.79.227.13:4803 , piece_id=EOOS6U6BBMXIY5C2V2JBRIDF7AA6W2CVGGTFGQWYTE76UHXOCNJQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.539479114 (322h26m35.090949575s) [noise=true , address=145.79.227.21:4473 , piece_id=LNYUOAJ72Y7JBDT46CKO3NJF5VI7BUMFRTHPOVQ37VPSVLMJJWNQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.541172168 (322h26m35.09264263s) [noise=true , address=145.79.227.16:4912 , piece_id=A5T4MWWZWMU7MZCQL5UALJ4NGZC4RRB5KGV65VXEPPX5LRU4PDEQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.541361829 (322h26m35.09283228s) [noise=true , address=145.79.227.9:4366 , piece_id=QUMSUINJ6LPOKUNBJH2SA3O52E74UBQ44VBY5UXGNYYHH56UMWHA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.543412674 (322h26m35.094883125s) [noise=true , address=145.79.227.8:4331 , piece_id=PLOUJIMJXBL3YQ2BFJDSTBZUWAZT65TVOSKEN3LAV6K4T2WOXCYA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.543625580 (322h26m35.095096041s) [noise=true , address=145.79.227.14:4839 , piece_id=U7AVK6SPVJ2YHCQ5JUMXFEYL7YXXOT43GB2SMFC3P3OQ7FJ34CZA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.544819604 (322h26m35.096290055s) [noise=true , address=145.79.227.11:4730 , piece_id=UILM3BBNJBLZXQIAM344ZWKAVNPZP5FOYV7BQFRO3K4X2GY3LT5A, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.545438793 (322h26m35.096909254s) [noise=true , address=77.90.31.21:4879 , piece_id=RC6YSH7QAQSOWU7CZFVBXST6IM625U743W5ARLTNWDAVZEAUS7HQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.546221353 (322h26m35.097691814s) [noise=true , address=145.79.227.32:4836 , piece_id=M42NYA2BQPW3CTOSCWSWCDFJ2YOOFLN3GMJEHIAQLF6FZ2UC3CHA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.547646487 (322h26m35.099116948s) [noise=true , address=45.91.116.5:4298 , piece_id=JIT6FAREK3TLHVJHJBUHBFHPSVFN6WV4XUQKGD3WNLG6O5SZWXNA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.548438445 (322h26m35.099908916s) [noise=true , address=145.79.227.22:4510 , piece_id=KQUMUDY2XK6WO654F6ZUU7QM5IHR2GAPQJYBPB2EBW2OI4MTDVGQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:38.320904919 (322h26m36.8723754s) [upload=283964]: Next returned nil segment Nov 22 18:30:38.320922753 (322h26m36.872393214s) [upload=283964]: Waiting for error group managing segments... Nov 22 18:30:38.649102998 (322h26m37.200573459s) [noise=true , address=45.91.116.4:4940 , piece_id=6LWBQSRMQYU7U3TUGUUMPDIOSX6AWCIO3RE46BXRS73YKR7E25DA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.664944633 (322h26m37.216415114s) [noise=true , address=45.91.116.5:4298 , piece_id=JIT6FAREK3TLHVJHJBUHBFHPSVFN6WV4XUQKGD3WNLG6O5SZWXNA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.667959895 (322h26m37.219430356s) [noise=true , address=77.90.31.13:4627 , piece_id=VJFT5OTJEZQYSPRL4QHAWFWF7HCRYMUZDESZO4BJM4NW5XZL7ZQQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.768748473 (322h26m37.320218954s) [noise=true , address=77.90.31.21:4879 , piece_id=RC6YSH7QAQSOWU7CZFVBXST6IM625U743W5ARLTNWDAVZEAUS7HQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:40.165076524 (322h26m38.716546975s) [noise=true , address=145.79.227.22:4510 , piece_id=KQUMUDY2XK6WO654F6ZUU7QM5IHR2GAPQJYBPB2EBW2OI4MTDVGQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:40.965617167 (322h26m39.517087629s) [noise=true , address=145.79.227.14:4839 , piece_id=U7AVK6SPVJ2YHCQ5JUMXFEYL7YXXOT43GB2SMFC3P3OQ7FJ34CZA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:42.701227963 (322h26m41.252698424s) [noise=true , address=145.79.227.9:4366 , piece_id=QUMSUINJ6LPOKUNBJH2SA3O52E74UBQ44VBY5UXGNYYHH56UMWHA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:43.663026451 (322h26m42.214496902s) [noise=true , address=145.79.227.13:4803 , piece_id=EOOS6U6BBMXIY5C2V2JBRIDF7AA6W2CVGGTFGQWYTE76UHXOCNJQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:45.482181600 (322h26m44.033652061s) [noise=true , address=145.79.227.20:4439 , piece_id=W3E226XLYQLNMGY2DSPI67AQNSF62Q6BD3GEOBYLK3VW7BQSIH3Q, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:48.700608997 (322h26m47.252079458s) [noise=true , address=145.79.227.21:4473 , piece_id=LNYUOAJ72Y7JBDT46CKO3NJF5VI7BUMFRTHPOVQ37VPSVLMJJWNQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:31:05.586366500 (322h27m4.137836971s) [noise=true , address=145.79.227.11:4730 , piece_id=UILM3BBNJBLZXQIAM344ZWKAVNPZP5FOYV7BQFRO3K4X2GY3LT5A, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection Nov 22 18:31:05.586387008 (322h27m4.137857469s) [noise=true , address=145.79.227.16:4912 , piece_id=A5T4MWWZWMU7MZCQL5UALJ4NGZC4RRB5KGV65VXEPPX5LRU4PDEQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection Nov 22 18:31:05.586537225 (322h27m4.138007686s) [noise=true , address=145.79.227.32:4836 , piece_id=M42NYA2BQPW3CTOSCWSWCDFJ2YOOFLN3GMJEHIAQLF6FZ2UC3CHA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection Nov 22 18:31:05.586630993 (322h27m4.138101454s) [noise=true , address=145.79.227.17:4970 , piece_id=7JXIJJROS4CCTDQOTOOL6XCNBA3ASBVGCO4YF75G5QRJ2QY2QYAA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection Nov 22 18:31:05.586783464 (322h27m4.138253915s) [noise=true , address=145.79.227.15:4875 , piece_id=HNFPLSNQQQM5M5I7J44WCAQL2XLRQ4GM44IHI7XI2ESNDCPFDWIQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection Nov 22 18:31:05.587158868 (322h27m4.138629349s) [noise=true , address=145.79.227.8:4331 , piece_id=PLOUJIMJXBL3YQ2BFJDSTBZUWAZT65TVOSKEN3LAV6K4T2WOXCYA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:37548->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 18:31:05.587230965 (322h27m4.138701426s) [seg_pos={0 0}, upload=283964]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 18:31:05.833640584 (322h27m4.385111055s) [upload=283964]: Done uploading segments... Nov 22 18:31:05.833691370 (322h27m4.385161841s) [upload=283964]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 5.83897457s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.24925508s trace: https://jaeger.dev.storj.io/trace/5225b8d0a62143ca FAIL (2025-11-22 18:31:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== s) [upload=283964]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 18:30:36.531047319 (322h26m35.08251778s) [seg_pos={0 0}, upload=283964]: Begin upload segment... Nov 22 18:30:36.531092805 (322h26m35.082563256s) [seg_pos={0 0}, upload=283964]: Done begin upload segment. Nov 22 18:30:36.531122301 (322h26m35.082592763s) [noise=true , address=77.90.31.13:4627 , piece_id=VJFT5OTJEZQYSPRL4QHAWFWF7HCRYMUZDESZO4BJM4NW5XZL7ZQQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.532650812 (322h26m35.084121263s) [noise=true , address=145.79.227.17:4970 , piece_id=7JXIJJROS4CCTDQOTOOL6XCNBA3ASBVGCO4YF75G5QRJ2QY2QYAA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.533640798 (322h26m35.085111259s) [noise=true , address=145.79.227.15:4875 , piece_id=HNFPLSNQQQM5M5I7J44WCAQL2XLRQ4GM44IHI7XI2ESNDCPFDWIQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.534514271 (322h26m35.085984732s) [noise=true , address=45.91.116.4:4940 , piece_id=6LWBQSRMQYU7U3TUGUUMPDIOSX6AWCIO3RE46BXRS73YKR7E25DA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.535280560 (322h26m35.086751021s) [noise=true , address=145.79.227.20:4439 , piece_id=W3E226XLYQLNMGY2DSPI67AQNSF62Q6BD3GEOBYLK3VW7BQSIH3Q, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.537679137 (322h26m35.089149599s) [noise=true , address=145.79.227.13:4803 , piece_id=EOOS6U6BBMXIY5C2V2JBRIDF7AA6W2CVGGTFGQWYTE76UHXOCNJQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.539479114 (322h26m35.090949575s) [noise=true , address=145.79.227.21:4473 , piece_id=LNYUOAJ72Y7JBDT46CKO3NJF5VI7BUMFRTHPOVQ37VPSVLMJJWNQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.541172168 (322h26m35.09264263s) [noise=true , address=145.79.227.16:4912 , piece_id=A5T4MWWZWMU7MZCQL5UALJ4NGZC4RRB5KGV65VXEPPX5LRU4PDEQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.541361829 (322h26m35.09283228s) [noise=true , address=145.79.227.9:4366 , piece_id=QUMSUINJ6LPOKUNBJH2SA3O52E74UBQ44VBY5UXGNYYHH56UMWHA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.543412674 (322h26m35.094883125s) [noise=true , address=145.79.227.8:4331 , piece_id=PLOUJIMJXBL3YQ2BFJDSTBZUWAZT65TVOSKEN3LAV6K4T2WOXCYA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.543625580 (322h26m35.095096041s) [noise=true , address=145.79.227.14:4839 , piece_id=U7AVK6SPVJ2YHCQ5JUMXFEYL7YXXOT43GB2SMFC3P3OQ7FJ34CZA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.544819604 (322h26m35.096290055s) [noise=true , address=145.79.227.11:4730 , piece_id=UILM3BBNJBLZXQIAM344ZWKAVNPZP5FOYV7BQFRO3K4X2GY3LT5A, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.545438793 (322h26m35.096909254s) [noise=true , address=77.90.31.21:4879 , piece_id=RC6YSH7QAQSOWU7CZFVBXST6IM625U743W5ARLTNWDAVZEAUS7HQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.546221353 (322h26m35.097691814s) [noise=true , address=145.79.227.32:4836 , piece_id=M42NYA2BQPW3CTOSCWSWCDFJ2YOOFLN3GMJEHIAQLF6FZ2UC3CHA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.547646487 (322h26m35.099116948s) [noise=true , address=45.91.116.5:4298 , piece_id=JIT6FAREK3TLHVJHJBUHBFHPSVFN6WV4XUQKGD3WNLG6O5SZWXNA, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:36.548438445 (322h26m35.099908916s) [noise=true , address=145.79.227.22:4510 , piece_id=KQUMUDY2XK6WO654F6ZUU7QM5IHR2GAPQJYBPB2EBW2OI4MTDVGQ, seg_pos={0 0}, upload=283964]: Uploading piece... Nov 22 18:30:38.320904919 (322h26m36.8723754s) [upload=283964]: Next returned nil segment Nov 22 18:30:38.320922753 (322h26m36.872393214s) [upload=283964]: Waiting for error group managing segments... Nov 22 18:30:38.649102998 (322h26m37.200573459s) [noise=true , address=45.91.116.4:4940 , piece_id=6LWBQSRMQYU7U3TUGUUMPDIOSX6AWCIO3RE46BXRS73YKR7E25DA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.664944633 (322h26m37.216415114s) [noise=true , address=45.91.116.5:4298 , piece_id=JIT6FAREK3TLHVJHJBUHBFHPSVFN6WV4XUQKGD3WNLG6O5SZWXNA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.667959895 (322h26m37.219430356s) [noise=true , address=77.90.31.13:4627 , piece_id=VJFT5OTJEZQYSPRL4QHAWFWF7HCRYMUZDESZO4BJM4NW5XZL7ZQQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:38.768748473 (322h26m37.320218954s) [noise=true , address=77.90.31.21:4879 , piece_id=RC6YSH7QAQSOWU7CZFVBXST6IM625U743W5ARLTNWDAVZEAUS7HQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:40.165076524 (322h26m38.716546975s) [noise=true , address=145.79.227.22:4510 , piece_id=KQUMUDY2XK6WO654F6ZUU7QM5IHR2GAPQJYBPB2EBW2OI4MTDVGQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:40.965617167 (322h26m39.517087629s) [noise=true , address=145.79.227.14:4839 , piece_id=U7AVK6SPVJ2YHCQ5JUMXFEYL7YXXOT43GB2SMFC3P3OQ7FJ34CZA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:42.701227963 (322h26m41.252698424s) [noise=true , address=145.79.227.9:4366 , piece_id=QUMSUINJ6LPOKUNBJH2SA3O52E74UBQ44VBY5UXGNYYHH56UMWHA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:43.663026451 (322h26m42.214496902s) [noise=true , address=145.79.227.13:4803 , piece_id=EOOS6U6BBMXIY5C2V2JBRIDF7AA6W2CVGGTFGQWYTE76UHXOCNJQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:45.482181600 (322h26m44.033652061s) [noise=true , address=145.79.227.20:4439 , piece_id=W3E226XLYQLNMGY2DSPI67AQNSF62Q6BD3GEOBYLK3VW7BQSIH3Q, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:30:48.700608997 (322h26m47.252079458s) [noise=true , address=145.79.227.21:4473 , piece_id=LNYUOAJ72Y7JBDT46CKO3NJF5VI7BUMFRTHPOVQ37VPSVLMJJWNQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: <nil> Nov 22 18:31:05.586366500 (322h27m4.137836971s) [noise=true , address=145.79.227.11:4730 , piece_id=UILM3BBNJBLZXQIAM344ZWKAVNPZP5FOYV7BQFRO3K4X2GY3LT5A, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34670->145.79.227.11:4730: use of closed network connection Nov 22 18:31:05.586387008 (322h27m4.137857469s) [noise=true , address=145.79.227.16:4912 , piece_id=A5T4MWWZWMU7MZCQL5UALJ4NGZC4RRB5KGV65VXEPPX5LRU4PDEQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56644->145.79.227.16:4912: use of closed network connection Nov 22 18:31:05.586537225 (322h27m4.138007686s) [noise=true , address=145.79.227.32:4836 , piece_id=M42NYA2BQPW3CTOSCWSWCDFJ2YOOFLN3GMJEHIAQLF6FZ2UC3CHA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44422->145.79.227.32:4836: use of closed network connection Nov 22 18:31:05.586630993 (322h27m4.138101454s) [noise=true , address=145.79.227.17:4970 , piece_id=7JXIJJROS4CCTDQOTOOL6XCNBA3ASBVGCO4YF75G5QRJ2QY2QYAA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55268->145.79.227.17:4970: use of closed network connection Nov 22 18:31:05.586783464 (322h27m4.138253915s) [noise=true , address=145.79.227.15:4875 , piece_id=HNFPLSNQQQM5M5I7J44WCAQL2XLRQ4GM44IHI7XI2ESNDCPFDWIQ, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60812->145.79.227.15:4875: use of closed network connection Nov 22 18:31:05.587158868 (322h27m4.138629349s) [noise=true , address=145.79.227.8:4331 , piece_id=PLOUJIMJXBL3YQ2BFJDSTBZUWAZT65TVOSKEN3LAV6K4T2WOXCYA, seg_pos={0 0}, upload=283964]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:37548->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 18:31:05.587230965 (322h27m4.138701426s) [seg_pos={0 0}, upload=283964]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 18:31:05.833640584 (322h27m4.385111055s) [upload=283964]: Done uploading segments... Nov 22 18:31:05.833691370 (322h27m4.385161841s) [upload=283964]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 30.249279517s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.439589356s trace: https://jaeger.dev.storj.io/trace/73a23c30a2f933a3 FAIL (2025-11-22 18:20:16 UTC) orbiter warning: context deadline exceeded Duration: 3.439621567s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.666617681s trace: https://jaeger.dev.storj.io/trace/73a23c30a2f933a3 FAIL (2025-11-22 18:19:13 UTC) orbiter warning: context deadline exceeded Duration: 3.666644241s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.524545039s trace: https://jaeger.dev.storj.io/trace/73a23c30a2f933a3 FAIL (2025-11-22 18:18:27 UTC) orbiter warning: context deadline exceeded Duration: 3.524617336s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.380777518s trace: https://jaeger.dev.storj.io/trace/73a23c30a2f933a3 FAIL (2025-11-22 18:18:04 UTC) orbiter warning: context deadline exceeded Duration: 3.380806645s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)