Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.933877245s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:30:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.933905539s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.90341157s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:30:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.903444262s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X.............X.........................................] duration: 5.000878619s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:30:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000897586s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X.............X........................................] duration: 3.457300162s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:29:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.457323688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.934759585s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:29:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.934788219s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X.............X.......................................] duration: 3.715877036s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:28:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.715904228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.90608423s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:28:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.90611525s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X......X.............X......................................] duration: 3.427613572s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:27:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.427745744s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.908267872s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:26:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.908298529s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X......X.............X.....................................] duration: 4.226813715s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:26:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.226848131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X......X.............X....................................] duration: 3.513995577s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:26:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.514021186s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.851472743s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:26:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.851501917s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.961800475s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:26:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.961832265s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X......X.............X...................................] duration: 3.784888172s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:25:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.784911327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.845805212s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:24:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.845821472s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X......X.............X..................................] duration: 3.430129656s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:24:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430157709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.930667593s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:23:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.930692662s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....X......X.............X.................................] duration: 3.410505166s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:23:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.41066999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.94645724s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:23:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.946511533s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.953644343s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:23:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.95366339s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X......X.............X................................] duration: 3.419652525s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:22:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.419713181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.912439013s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:22:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.91246975s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......X......X.............X...............................] duration: 3.484125119s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:21:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484148243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.941223664s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:21:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.941247509s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........X......X.............X..............................] duration: 3.695007837s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:20:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.695039647s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........X......X.............X.............................] duration: 3.699258184s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:19:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.699287351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 11.174975457s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:19:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.175004973s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.841326644s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:19:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.841355649s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........X......X.............X............................] duration: 3.480276081s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:18:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.480294314s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........X......X.............X...........................] duration: 3.878456615s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:18:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.878481623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.098464285s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:18:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.098479134s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............X......X.............X..........................] duration: 4.138722907s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:18:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.138752202s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.913971383s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:17:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.914014907s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X......X.............X.........................] duration: 3.57876116s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:16:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.578789915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.880545944s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:16:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.880571452s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X.............X......X.............X........................] duration: 3.540137347s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:16:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.540263357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.977227923s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:16:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.977257569s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X.............X......X.............X.......................] duration: 3.559191215s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:15:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559223597s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X.............X......X.............X......................] duration: 4.031619444s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:15:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.031650343s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.953064652s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:15:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.95309482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.82275855s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:15:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.822797223s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X.............X......X.............X.....................] duration: 4.195644483s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:15:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.195675733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.967915008s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:14:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.967943821s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X.............X......X.............X....................] duration: 4.179822015s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:14:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.179904221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.852799928s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:13:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.852839903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.969664978s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:13:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.969694864s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X.............X......X.............X...................] duration: 4.195855253s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:12:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.195922271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......X.............X......X.............X..................] duration: 4.318582915s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:12:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.318616859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.955289696s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:12:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.955337238s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......X.............X......X.............X.................] duration: 4.653245133s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:11:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.653277153s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)