Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.953064652s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:15:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.95309482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.82275855s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:15:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.822797223s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X.............X......X.............X.....................] duration: 4.195644483s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:15:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.195675733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.967915008s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:14:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.967943821s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X.............X......X.............X....................] duration: 4.179822015s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:14:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.179904221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.852799928s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:13:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.852839903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.969664978s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:13:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.969694864s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X.............X......X.............X...................] duration: 4.195855253s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:12:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.195922271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......X.............X......X.............X..................] duration: 4.318582915s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:12:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.318616859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.955289696s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:12:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.955337238s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......X.............X......X.............X.................] duration: 4.653245133s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:11:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.653277153s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.990076907s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:11:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.990110841s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........X.............X......X.............X................] duration: 4.612002765s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:11:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.612093737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.72578745s trace: https://jaeger.dev.storj.io/trace/5860d9055b13d3d5 FAIL (2025-11-21 03:11:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02:25:33.629924468 (282h21m32.181394939s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630349578 (282h21m32.181820039s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630843158 (282h21m32.182313619s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.631501040 (282h21m32.182971502s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.632877661 (282h21m32.184348112s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.633957257 (282h21m32.185427718s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635008760 (282h21m32.186479221s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635936546 (282h21m32.187407007s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.636636048 (282h21m32.188106509s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:34.461042315 (282h21m33.012512776s) [upload=248651]: Next returned nil segment Nov 21 02:25:34.461079235 (282h21m33.012549697s) [upload=248651]: Waiting for error group managing segments... Nov 21 02:25:34.850359259 (282h21m33.40182972s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.852076179 (282h21m33.40354664s) [noise=true , address=185.167.204.206:4055 , piece_id=H54UI2MK43VZPELPTCU6XNLESYE6S5XZ4ZKAOV66TP3BBKKKDMFA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.856988012 (282h21m33.408458473s) [noise=true , address=185.167.204.205:4031 , piece_id=MCKJTCBSRINYPN27XNNFMXYPDHE3KVMSCWOB7VCOETNIO4F3PZVQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858141178 (282h21m33.409611639s) [noise=true , address=185.167.204.206:4067 , piece_id=TQD5JG4TGZYKCG4KOAWEUF6BSVY2IX35O2IB6VYKKH7ABNQEMRRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858405160 (282h21m33.409875621s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858612054 (282h21m33.410082515s) [noise=true , address=185.167.204.205:4023 , piece_id=NQMCNM7GAS2URY7JNN4DXMU2FZ7UAVHC42U55IVIGYUWXZ5DFOCQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859019680 (282h21m33.410490131s) [noise=true , address=185.167.204.205:4013 , piece_id=G5AFIVUYYIG3IAVQ2UN4MTL3NPXN326IOR77CXTU4BAIBFMICKXQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859464347 (282h21m33.410934808s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859565901 (282h21m33.411036352s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866136853 (282h21m33.417607324s) [noise=true , address=185.167.204.205:4027 , piece_id=DTF2OMEXXT2XFQB2RQOKL66K4AOIC3DTZOTE2ZGIMYWVPCTKJWJA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866678734 (282h21m33.418149195s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867348910 (282h21m33.418819361s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867982197 (282h21m33.419452668s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.868482399 (282h21m33.41995286s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.880010110 (282h21m33.431480601s) [noise=true , address=185.167.204.205:4025 , piece_id=ZWXX3V2QD4QSVDNMMDJ5GSSLIMSAK5IZOHOEB2SI3TFMPUXRRVXA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.419467556 (282h21m33.970938017s) [noise=true , address=81.171.14.70:4044 , piece_id=WAQ3SBHX6TRISFEOPE4Y6O2HSE2NYVC257V6ROMZSFVJO7YLCGRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.424133200 (282h21m33.975603661s) [noise=true , address=81.171.14.70:4038 , piece_id=X67ZAHX3AIYBBWES2PL5TB6FKR3DGZBALSLTITODERBWUI2OILVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.433560922 (282h21m33.985031383s) [noise=true , address=81.171.14.70:4041 , piece_id=AAUJYA6S7AWTCUO33QBNLCFUKRLGY6NLGFHRECJ5I63H53J46SMQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.452011215 (282h21m34.003481676s) [noise=true , address=81.171.14.70:4047 , piece_id=ZNKZHAJAP24ZF6ZBV6762YY7LCRUSQCCL5E53WU5TMZ457CRZ3RA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.508011640 (282h21m34.059482111s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.897112773 (282h21m34.448583234s) [noise=true , address=81.171.14.70:4031 , piece_id=LYWET76RCJLTUN3DJDSVMDOZ6X5S53KPZ264P7D5FS7XCRDSYIQA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.027285292 (282h21m34.578755743s) [noise=true , address=81.171.14.70:4045 , piece_id=EOJUCJVNYJIF447J6FKIRVIXM3IJ5OU6TEVUKHFKQCUUVIMBR2EA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.070911284 (282h21m34.622381745s) [noise=true , address=89.149.201.147:4011 , piece_id=F5544PH4I4Z224P2DVMAXLM57DWIFT37KKXUJBR7B3OG5FV7KK6A, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.073062550 (282h21m34.624533011s) [noise=true , address=89.149.201.147:4018 , piece_id=LPC2AQ7TX4UUEI4I5VMGDOIFJI7R3GJGAOL5Q2O7SVY42OFN3XVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.083744591 (282h21m34.635215052s) [noise=true , address=89.149.201.147:4008 , piece_id=GSCXEJOMPWFECP7YSD66BRY26ZTXGIRZIMZYWKIW2SLRGPSJT7OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.146992305 (282h21m34.698462766s) [noise=true , address=89.149.201.147:4003 , piece_id=SDOIACESWFHPNPOZUZIXW5F2K62OUKCLCBHUKD4YKPLUQG4XXE5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.147018144 (282h21m34.698488605s) [seg_pos={0 0}, upload=248651]: Segment reached optimal threshold of 26 pieces. Nov 21 02:25:36.147392256 (282h21m34.698862717s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Nov 21 02:25:36.147403267 (282h21m34.698873728s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148140441 (282h21m34.699610933s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9): protocol: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection Nov 21 02:25:36.148153566 (282h21m34.699624027s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148213310 (282h21m34.699683761s) [seg_pos={0 0}, upload=248651]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 02:25:36.148357414 (282h21m34.699827875s) [upload=248651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 02:25:36.148395076 (282h21m34.699865547s) [upload=248651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 02:25:36.406626090 (282h21m34.958096551s) [upload=248651]: Done uploading segments... Nov 21 02:25:36.406651539 (282h21m34.958122s) [upload=248651]: Upload finished. err: <nil> Duration: 5.725821174s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.970310342s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:10:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.970341771s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........X.............X......X.............X...............] duration: 3.584739075s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:10:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.584769373s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.227607031s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:10:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.227632849s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.620285837s trace: https://jaeger.dev.storj.io/trace/5860d9055b13d3d5 FAIL (2025-11-21 03:10:03 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02:25:33.629924468 (282h21m32.181394939s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630349578 (282h21m32.181820039s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630843158 (282h21m32.182313619s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.631501040 (282h21m32.182971502s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.632877661 (282h21m32.184348112s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.633957257 (282h21m32.185427718s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635008760 (282h21m32.186479221s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635936546 (282h21m32.187407007s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.636636048 (282h21m32.188106509s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:34.461042315 (282h21m33.012512776s) [upload=248651]: Next returned nil segment Nov 21 02:25:34.461079235 (282h21m33.012549697s) [upload=248651]: Waiting for error group managing segments... Nov 21 02:25:34.850359259 (282h21m33.40182972s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.852076179 (282h21m33.40354664s) [noise=true , address=185.167.204.206:4055 , piece_id=H54UI2MK43VZPELPTCU6XNLESYE6S5XZ4ZKAOV66TP3BBKKKDMFA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.856988012 (282h21m33.408458473s) [noise=true , address=185.167.204.205:4031 , piece_id=MCKJTCBSRINYPN27XNNFMXYPDHE3KVMSCWOB7VCOETNIO4F3PZVQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858141178 (282h21m33.409611639s) [noise=true , address=185.167.204.206:4067 , piece_id=TQD5JG4TGZYKCG4KOAWEUF6BSVY2IX35O2IB6VYKKH7ABNQEMRRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858405160 (282h21m33.409875621s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858612054 (282h21m33.410082515s) [noise=true , address=185.167.204.205:4023 , piece_id=NQMCNM7GAS2URY7JNN4DXMU2FZ7UAVHC42U55IVIGYUWXZ5DFOCQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859019680 (282h21m33.410490131s) [noise=true , address=185.167.204.205:4013 , piece_id=G5AFIVUYYIG3IAVQ2UN4MTL3NPXN326IOR77CXTU4BAIBFMICKXQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859464347 (282h21m33.410934808s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859565901 (282h21m33.411036352s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866136853 (282h21m33.417607324s) [noise=true , address=185.167.204.205:4027 , piece_id=DTF2OMEXXT2XFQB2RQOKL66K4AOIC3DTZOTE2ZGIMYWVPCTKJWJA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866678734 (282h21m33.418149195s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867348910 (282h21m33.418819361s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867982197 (282h21m33.419452668s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.868482399 (282h21m33.41995286s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.880010110 (282h21m33.431480601s) [noise=true , address=185.167.204.205:4025 , piece_id=ZWXX3V2QD4QSVDNMMDJ5GSSLIMSAK5IZOHOEB2SI3TFMPUXRRVXA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.419467556 (282h21m33.970938017s) [noise=true , address=81.171.14.70:4044 , piece_id=WAQ3SBHX6TRISFEOPE4Y6O2HSE2NYVC257V6ROMZSFVJO7YLCGRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.424133200 (282h21m33.975603661s) [noise=true , address=81.171.14.70:4038 , piece_id=X67ZAHX3AIYBBWES2PL5TB6FKR3DGZBALSLTITODERBWUI2OILVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.433560922 (282h21m33.985031383s) [noise=true , address=81.171.14.70:4041 , piece_id=AAUJYA6S7AWTCUO33QBNLCFUKRLGY6NLGFHRECJ5I63H53J46SMQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.452011215 (282h21m34.003481676s) [noise=true , address=81.171.14.70:4047 , piece_id=ZNKZHAJAP24ZF6ZBV6762YY7LCRUSQCCL5E53WU5TMZ457CRZ3RA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.508011640 (282h21m34.059482111s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.897112773 (282h21m34.448583234s) [noise=true , address=81.171.14.70:4031 , piece_id=LYWET76RCJLTUN3DJDSVMDOZ6X5S53KPZ264P7D5FS7XCRDSYIQA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.027285292 (282h21m34.578755743s) [noise=true , address=81.171.14.70:4045 , piece_id=EOJUCJVNYJIF447J6FKIRVIXM3IJ5OU6TEVUKHFKQCUUVIMBR2EA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.070911284 (282h21m34.622381745s) [noise=true , address=89.149.201.147:4011 , piece_id=F5544PH4I4Z224P2DVMAXLM57DWIFT37KKXUJBR7B3OG5FV7KK6A, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.073062550 (282h21m34.624533011s) [noise=true , address=89.149.201.147:4018 , piece_id=LPC2AQ7TX4UUEI4I5VMGDOIFJI7R3GJGAOL5Q2O7SVY42OFN3XVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.083744591 (282h21m34.635215052s) [noise=true , address=89.149.201.147:4008 , piece_id=GSCXEJOMPWFECP7YSD66BRY26ZTXGIRZIMZYWKIW2SLRGPSJT7OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.146992305 (282h21m34.698462766s) [noise=true , address=89.149.201.147:4003 , piece_id=SDOIACESWFHPNPOZUZIXW5F2K62OUKCLCBHUKD4YKPLUQG4XXE5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.147018144 (282h21m34.698488605s) [seg_pos={0 0}, upload=248651]: Segment reached optimal threshold of 26 pieces. Nov 21 02:25:36.147392256 (282h21m34.698862717s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Nov 21 02:25:36.147403267 (282h21m34.698873728s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148140441 (282h21m34.699610933s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9): protocol: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection Nov 21 02:25:36.148153566 (282h21m34.699624027s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148213310 (282h21m34.699683761s) [seg_pos={0 0}, upload=248651]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 02:25:36.148357414 (282h21m34.699827875s) [upload=248651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 02:25:36.148395076 (282h21m34.699865547s) [upload=248651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 02:25:36.406626090 (282h21m34.958096551s) [upload=248651]: Done uploading segments... Nov 21 02:25:36.406651539 (282h21m34.958122s) [upload=248651]: Upload finished. err: <nil> Duration: 5.620316396s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........X.............X......X.............X..............] duration: 3.497491154s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:09:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497520841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.262504097s trace: https://jaeger.dev.storj.io/trace/5860d9055b13d3d5 FAIL (2025-11-21 03:09:05 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02:25:33.629924468 (282h21m32.181394939s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630349578 (282h21m32.181820039s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630843158 (282h21m32.182313619s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.631501040 (282h21m32.182971502s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.632877661 (282h21m32.184348112s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.633957257 (282h21m32.185427718s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635008760 (282h21m32.186479221s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635936546 (282h21m32.187407007s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.636636048 (282h21m32.188106509s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:34.461042315 (282h21m33.012512776s) [upload=248651]: Next returned nil segment Nov 21 02:25:34.461079235 (282h21m33.012549697s) [upload=248651]: Waiting for error group managing segments... Nov 21 02:25:34.850359259 (282h21m33.40182972s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.852076179 (282h21m33.40354664s) [noise=true , address=185.167.204.206:4055 , piece_id=H54UI2MK43VZPELPTCU6XNLESYE6S5XZ4ZKAOV66TP3BBKKKDMFA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.856988012 (282h21m33.408458473s) [noise=true , address=185.167.204.205:4031 , piece_id=MCKJTCBSRINYPN27XNNFMXYPDHE3KVMSCWOB7VCOETNIO4F3PZVQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858141178 (282h21m33.409611639s) [noise=true , address=185.167.204.206:4067 , piece_id=TQD5JG4TGZYKCG4KOAWEUF6BSVY2IX35O2IB6VYKKH7ABNQEMRRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858405160 (282h21m33.409875621s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858612054 (282h21m33.410082515s) [noise=true , address=185.167.204.205:4023 , piece_id=NQMCNM7GAS2URY7JNN4DXMU2FZ7UAVHC42U55IVIGYUWXZ5DFOCQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859019680 (282h21m33.410490131s) [noise=true , address=185.167.204.205:4013 , piece_id=G5AFIVUYYIG3IAVQ2UN4MTL3NPXN326IOR77CXTU4BAIBFMICKXQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859464347 (282h21m33.410934808s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859565901 (282h21m33.411036352s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866136853 (282h21m33.417607324s) [noise=true , address=185.167.204.205:4027 , piece_id=DTF2OMEXXT2XFQB2RQOKL66K4AOIC3DTZOTE2ZGIMYWVPCTKJWJA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866678734 (282h21m33.418149195s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867348910 (282h21m33.418819361s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867982197 (282h21m33.419452668s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.868482399 (282h21m33.41995286s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.880010110 (282h21m33.431480601s) [noise=true , address=185.167.204.205:4025 , piece_id=ZWXX3V2QD4QSVDNMMDJ5GSSLIMSAK5IZOHOEB2SI3TFMPUXRRVXA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.419467556 (282h21m33.970938017s) [noise=true , address=81.171.14.70:4044 , piece_id=WAQ3SBHX6TRISFEOPE4Y6O2HSE2NYVC257V6ROMZSFVJO7YLCGRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.424133200 (282h21m33.975603661s) [noise=true , address=81.171.14.70:4038 , piece_id=X67ZAHX3AIYBBWES2PL5TB6FKR3DGZBALSLTITODERBWUI2OILVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.433560922 (282h21m33.985031383s) [noise=true , address=81.171.14.70:4041 , piece_id=AAUJYA6S7AWTCUO33QBNLCFUKRLGY6NLGFHRECJ5I63H53J46SMQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.452011215 (282h21m34.003481676s) [noise=true , address=81.171.14.70:4047 , piece_id=ZNKZHAJAP24ZF6ZBV6762YY7LCRUSQCCL5E53WU5TMZ457CRZ3RA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.508011640 (282h21m34.059482111s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.897112773 (282h21m34.448583234s) [noise=true , address=81.171.14.70:4031 , piece_id=LYWET76RCJLTUN3DJDSVMDOZ6X5S53KPZ264P7D5FS7XCRDSYIQA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.027285292 (282h21m34.578755743s) [noise=true , address=81.171.14.70:4045 , piece_id=EOJUCJVNYJIF447J6FKIRVIXM3IJ5OU6TEVUKHFKQCUUVIMBR2EA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.070911284 (282h21m34.622381745s) [noise=true , address=89.149.201.147:4011 , piece_id=F5544PH4I4Z224P2DVMAXLM57DWIFT37KKXUJBR7B3OG5FV7KK6A, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.073062550 (282h21m34.624533011s) [noise=true , address=89.149.201.147:4018 , piece_id=LPC2AQ7TX4UUEI4I5VMGDOIFJI7R3GJGAOL5Q2O7SVY42OFN3XVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.083744591 (282h21m34.635215052s) [noise=true , address=89.149.201.147:4008 , piece_id=GSCXEJOMPWFECP7YSD66BRY26ZTXGIRZIMZYWKIW2SLRGPSJT7OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.146992305 (282h21m34.698462766s) [noise=true , address=89.149.201.147:4003 , piece_id=SDOIACESWFHPNPOZUZIXW5F2K62OUKCLCBHUKD4YKPLUQG4XXE5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.147018144 (282h21m34.698488605s) [seg_pos={0 0}, upload=248651]: Segment reached optimal threshold of 26 pieces. Nov 21 02:25:36.147392256 (282h21m34.698862717s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Nov 21 02:25:36.147403267 (282h21m34.698873728s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148140441 (282h21m34.699610933s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9): protocol: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection Nov 21 02:25:36.148153566 (282h21m34.699624027s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148213310 (282h21m34.699683761s) [seg_pos={0 0}, upload=248651]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 02:25:36.148357414 (282h21m34.699827875s) [upload=248651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 02:25:36.148395076 (282h21m34.699865547s) [upload=248651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 02:25:36.406626090 (282h21m34.958096551s) [upload=248651]: Done uploading segments... Nov 21 02:25:36.406651539 (282h21m34.958122s) [upload=248651]: Upload finished. err: <nil> Duration: 6.26253754s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.977564469s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:09:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.977625886s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........X.............X......X.............X.............] duration: 3.775133634s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:09:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.775445357s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............X.............X......X.............X............] duration: 3.810280582s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:08:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.810310578s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.768035416s trace: https://jaeger.dev.storj.io/trace/5860d9055b13d3d5 FAIL (2025-11-21 03:08:30 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02:25:33.629924468 (282h21m32.181394939s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630349578 (282h21m32.181820039s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630843158 (282h21m32.182313619s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.631501040 (282h21m32.182971502s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.632877661 (282h21m32.184348112s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.633957257 (282h21m32.185427718s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635008760 (282h21m32.186479221s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635936546 (282h21m32.187407007s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.636636048 (282h21m32.188106509s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:34.461042315 (282h21m33.012512776s) [upload=248651]: Next returned nil segment Nov 21 02:25:34.461079235 (282h21m33.012549697s) [upload=248651]: Waiting for error group managing segments... Nov 21 02:25:34.850359259 (282h21m33.40182972s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.852076179 (282h21m33.40354664s) [noise=true , address=185.167.204.206:4055 , piece_id=H54UI2MK43VZPELPTCU6XNLESYE6S5XZ4ZKAOV66TP3BBKKKDMFA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.856988012 (282h21m33.408458473s) [noise=true , address=185.167.204.205:4031 , piece_id=MCKJTCBSRINYPN27XNNFMXYPDHE3KVMSCWOB7VCOETNIO4F3PZVQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858141178 (282h21m33.409611639s) [noise=true , address=185.167.204.206:4067 , piece_id=TQD5JG4TGZYKCG4KOAWEUF6BSVY2IX35O2IB6VYKKH7ABNQEMRRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858405160 (282h21m33.409875621s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858612054 (282h21m33.410082515s) [noise=true , address=185.167.204.205:4023 , piece_id=NQMCNM7GAS2URY7JNN4DXMU2FZ7UAVHC42U55IVIGYUWXZ5DFOCQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859019680 (282h21m33.410490131s) [noise=true , address=185.167.204.205:4013 , piece_id=G5AFIVUYYIG3IAVQ2UN4MTL3NPXN326IOR77CXTU4BAIBFMICKXQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859464347 (282h21m33.410934808s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859565901 (282h21m33.411036352s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866136853 (282h21m33.417607324s) [noise=true , address=185.167.204.205:4027 , piece_id=DTF2OMEXXT2XFQB2RQOKL66K4AOIC3DTZOTE2ZGIMYWVPCTKJWJA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866678734 (282h21m33.418149195s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867348910 (282h21m33.418819361s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867982197 (282h21m33.419452668s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.868482399 (282h21m33.41995286s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.880010110 (282h21m33.431480601s) [noise=true , address=185.167.204.205:4025 , piece_id=ZWXX3V2QD4QSVDNMMDJ5GSSLIMSAK5IZOHOEB2SI3TFMPUXRRVXA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.419467556 (282h21m33.970938017s) [noise=true , address=81.171.14.70:4044 , piece_id=WAQ3SBHX6TRISFEOPE4Y6O2HSE2NYVC257V6ROMZSFVJO7YLCGRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.424133200 (282h21m33.975603661s) [noise=true , address=81.171.14.70:4038 , piece_id=X67ZAHX3AIYBBWES2PL5TB6FKR3DGZBALSLTITODERBWUI2OILVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.433560922 (282h21m33.985031383s) [noise=true , address=81.171.14.70:4041 , piece_id=AAUJYA6S7AWTCUO33QBNLCFUKRLGY6NLGFHRECJ5I63H53J46SMQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.452011215 (282h21m34.003481676s) [noise=true , address=81.171.14.70:4047 , piece_id=ZNKZHAJAP24ZF6ZBV6762YY7LCRUSQCCL5E53WU5TMZ457CRZ3RA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.508011640 (282h21m34.059482111s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.897112773 (282h21m34.448583234s) [noise=true , address=81.171.14.70:4031 , piece_id=LYWET76RCJLTUN3DJDSVMDOZ6X5S53KPZ264P7D5FS7XCRDSYIQA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.027285292 (282h21m34.578755743s) [noise=true , address=81.171.14.70:4045 , piece_id=EOJUCJVNYJIF447J6FKIRVIXM3IJ5OU6TEVUKHFKQCUUVIMBR2EA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.070911284 (282h21m34.622381745s) [noise=true , address=89.149.201.147:4011 , piece_id=F5544PH4I4Z224P2DVMAXLM57DWIFT37KKXUJBR7B3OG5FV7KK6A, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.073062550 (282h21m34.624533011s) [noise=true , address=89.149.201.147:4018 , piece_id=LPC2AQ7TX4UUEI4I5VMGDOIFJI7R3GJGAOL5Q2O7SVY42OFN3XVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.083744591 (282h21m34.635215052s) [noise=true , address=89.149.201.147:4008 , piece_id=GSCXEJOMPWFECP7YSD66BRY26ZTXGIRZIMZYWKIW2SLRGPSJT7OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.146992305 (282h21m34.698462766s) [noise=true , address=89.149.201.147:4003 , piece_id=SDOIACESWFHPNPOZUZIXW5F2K62OUKCLCBHUKD4YKPLUQG4XXE5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.147018144 (282h21m34.698488605s) [seg_pos={0 0}, upload=248651]: Segment reached optimal threshold of 26 pieces. Nov 21 02:25:36.147392256 (282h21m34.698862717s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Nov 21 02:25:36.147403267 (282h21m34.698873728s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148140441 (282h21m34.699610933s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9): protocol: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection Nov 21 02:25:36.148153566 (282h21m34.699624027s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148213310 (282h21m34.699683761s) [seg_pos={0 0}, upload=248651]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 02:25:36.148357414 (282h21m34.699827875s) [upload=248651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 02:25:36.148395076 (282h21m34.699865547s) [upload=248651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 02:25:36.406626090 (282h21m34.958096551s) [upload=248651]: Done uploading segments... Nov 21 02:25:36.406651539 (282h21m34.958122s) [upload=248651]: Upload finished. err: <nil> Duration: 5.76806371s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.904210966s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:08:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.904316147s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.915788811s trace: https://jaeger.dev.storj.io/trace/5860d9055b13d3d5 FAIL (2025-11-21 03:08:11 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02:25:33.629924468 (282h21m32.181394939s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630349578 (282h21m32.181820039s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630843158 (282h21m32.182313619s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.631501040 (282h21m32.182971502s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.632877661 (282h21m32.184348112s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.633957257 (282h21m32.185427718s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635008760 (282h21m32.186479221s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635936546 (282h21m32.187407007s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.636636048 (282h21m32.188106509s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:34.461042315 (282h21m33.012512776s) [upload=248651]: Next returned nil segment Nov 21 02:25:34.461079235 (282h21m33.012549697s) [upload=248651]: Waiting for error group managing segments... Nov 21 02:25:34.850359259 (282h21m33.40182972s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.852076179 (282h21m33.40354664s) [noise=true , address=185.167.204.206:4055 , piece_id=H54UI2MK43VZPELPTCU6XNLESYE6S5XZ4ZKAOV66TP3BBKKKDMFA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.856988012 (282h21m33.408458473s) [noise=true , address=185.167.204.205:4031 , piece_id=MCKJTCBSRINYPN27XNNFMXYPDHE3KVMSCWOB7VCOETNIO4F3PZVQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858141178 (282h21m33.409611639s) [noise=true , address=185.167.204.206:4067 , piece_id=TQD5JG4TGZYKCG4KOAWEUF6BSVY2IX35O2IB6VYKKH7ABNQEMRRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858405160 (282h21m33.409875621s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858612054 (282h21m33.410082515s) [noise=true , address=185.167.204.205:4023 , piece_id=NQMCNM7GAS2URY7JNN4DXMU2FZ7UAVHC42U55IVIGYUWXZ5DFOCQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859019680 (282h21m33.410490131s) [noise=true , address=185.167.204.205:4013 , piece_id=G5AFIVUYYIG3IAVQ2UN4MTL3NPXN326IOR77CXTU4BAIBFMICKXQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859464347 (282h21m33.410934808s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859565901 (282h21m33.411036352s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866136853 (282h21m33.417607324s) [noise=true , address=185.167.204.205:4027 , piece_id=DTF2OMEXXT2XFQB2RQOKL66K4AOIC3DTZOTE2ZGIMYWVPCTKJWJA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866678734 (282h21m33.418149195s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867348910 (282h21m33.418819361s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867982197 (282h21m33.419452668s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.868482399 (282h21m33.41995286s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.880010110 (282h21m33.431480601s) [noise=true , address=185.167.204.205:4025 , piece_id=ZWXX3V2QD4QSVDNMMDJ5GSSLIMSAK5IZOHOEB2SI3TFMPUXRRVXA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.419467556 (282h21m33.970938017s) [noise=true , address=81.171.14.70:4044 , piece_id=WAQ3SBHX6TRISFEOPE4Y6O2HSE2NYVC257V6ROMZSFVJO7YLCGRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.424133200 (282h21m33.975603661s) [noise=true , address=81.171.14.70:4038 , piece_id=X67ZAHX3AIYBBWES2PL5TB6FKR3DGZBALSLTITODERBWUI2OILVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.433560922 (282h21m33.985031383s) [noise=true , address=81.171.14.70:4041 , piece_id=AAUJYA6S7AWTCUO33QBNLCFUKRLGY6NLGFHRECJ5I63H53J46SMQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.452011215 (282h21m34.003481676s) [noise=true , address=81.171.14.70:4047 , piece_id=ZNKZHAJAP24ZF6ZBV6762YY7LCRUSQCCL5E53WU5TMZ457CRZ3RA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.508011640 (282h21m34.059482111s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.897112773 (282h21m34.448583234s) [noise=true , address=81.171.14.70:4031 , piece_id=LYWET76RCJLTUN3DJDSVMDOZ6X5S53KPZ264P7D5FS7XCRDSYIQA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.027285292 (282h21m34.578755743s) [noise=true , address=81.171.14.70:4045 , piece_id=EOJUCJVNYJIF447J6FKIRVIXM3IJ5OU6TEVUKHFKQCUUVIMBR2EA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.070911284 (282h21m34.622381745s) [noise=true , address=89.149.201.147:4011 , piece_id=F5544PH4I4Z224P2DVMAXLM57DWIFT37KKXUJBR7B3OG5FV7KK6A, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.073062550 (282h21m34.624533011s) [noise=true , address=89.149.201.147:4018 , piece_id=LPC2AQ7TX4UUEI4I5VMGDOIFJI7R3GJGAOL5Q2O7SVY42OFN3XVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.083744591 (282h21m34.635215052s) [noise=true , address=89.149.201.147:4008 , piece_id=GSCXEJOMPWFECP7YSD66BRY26ZTXGIRZIMZYWKIW2SLRGPSJT7OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.146992305 (282h21m34.698462766s) [noise=true , address=89.149.201.147:4003 , piece_id=SDOIACESWFHPNPOZUZIXW5F2K62OUKCLCBHUKD4YKPLUQG4XXE5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.147018144 (282h21m34.698488605s) [seg_pos={0 0}, upload=248651]: Segment reached optimal threshold of 26 pieces. Nov 21 02:25:36.147392256 (282h21m34.698862717s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Nov 21 02:25:36.147403267 (282h21m34.698873728s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148140441 (282h21m34.699610933s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9): protocol: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection Nov 21 02:25:36.148153566 (282h21m34.699624027s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148213310 (282h21m34.699683761s) [seg_pos={0 0}, upload=248651]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 02:25:36.148357414 (282h21m34.699827875s) [upload=248651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 02:25:36.148395076 (282h21m34.699865547s) [upload=248651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 02:25:36.406626090 (282h21m34.958096551s) [upload=248651]: Done uploading segments... Nov 21 02:25:36.406651539 (282h21m34.958122s) [upload=248651]: Upload finished. err: <nil> Duration: 5.915846471s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.950245901s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:07:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.950269396s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............X.............X......X.............X...........] duration: 4.532946018s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:07:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.532976926s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.89511132s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:07:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.895145985s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.285308993s trace: https://jaeger.dev.storj.io/trace/5860d9055b13d3d5 FAIL (2025-11-21 03:07:05 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02:25:33.629924468 (282h21m32.181394939s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630349578 (282h21m32.181820039s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630843158 (282h21m32.182313619s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.631501040 (282h21m32.182971502s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.632877661 (282h21m32.184348112s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.633957257 (282h21m32.185427718s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635008760 (282h21m32.186479221s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635936546 (282h21m32.187407007s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.636636048 (282h21m32.188106509s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:34.461042315 (282h21m33.012512776s) [upload=248651]: Next returned nil segment Nov 21 02:25:34.461079235 (282h21m33.012549697s) [upload=248651]: Waiting for error group managing segments... Nov 21 02:25:34.850359259 (282h21m33.40182972s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.852076179 (282h21m33.40354664s) [noise=true , address=185.167.204.206:4055 , piece_id=H54UI2MK43VZPELPTCU6XNLESYE6S5XZ4ZKAOV66TP3BBKKKDMFA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.856988012 (282h21m33.408458473s) [noise=true , address=185.167.204.205:4031 , piece_id=MCKJTCBSRINYPN27XNNFMXYPDHE3KVMSCWOB7VCOETNIO4F3PZVQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858141178 (282h21m33.409611639s) [noise=true , address=185.167.204.206:4067 , piece_id=TQD5JG4TGZYKCG4KOAWEUF6BSVY2IX35O2IB6VYKKH7ABNQEMRRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858405160 (282h21m33.409875621s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858612054 (282h21m33.410082515s) [noise=true , address=185.167.204.205:4023 , piece_id=NQMCNM7GAS2URY7JNN4DXMU2FZ7UAVHC42U55IVIGYUWXZ5DFOCQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859019680 (282h21m33.410490131s) [noise=true , address=185.167.204.205:4013 , piece_id=G5AFIVUYYIG3IAVQ2UN4MTL3NPXN326IOR77CXTU4BAIBFMICKXQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859464347 (282h21m33.410934808s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859565901 (282h21m33.411036352s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866136853 (282h21m33.417607324s) [noise=true , address=185.167.204.205:4027 , piece_id=DTF2OMEXXT2XFQB2RQOKL66K4AOIC3DTZOTE2ZGIMYWVPCTKJWJA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866678734 (282h21m33.418149195s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867348910 (282h21m33.418819361s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867982197 (282h21m33.419452668s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.868482399 (282h21m33.41995286s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.880010110 (282h21m33.431480601s) [noise=true , address=185.167.204.205:4025 , piece_id=ZWXX3V2QD4QSVDNMMDJ5GSSLIMSAK5IZOHOEB2SI3TFMPUXRRVXA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.419467556 (282h21m33.970938017s) [noise=true , address=81.171.14.70:4044 , piece_id=WAQ3SBHX6TRISFEOPE4Y6O2HSE2NYVC257V6ROMZSFVJO7YLCGRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.424133200 (282h21m33.975603661s) [noise=true , address=81.171.14.70:4038 , piece_id=X67ZAHX3AIYBBWES2PL5TB6FKR3DGZBALSLTITODERBWUI2OILVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.433560922 (282h21m33.985031383s) [noise=true , address=81.171.14.70:4041 , piece_id=AAUJYA6S7AWTCUO33QBNLCFUKRLGY6NLGFHRECJ5I63H53J46SMQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.452011215 (282h21m34.003481676s) [noise=true , address=81.171.14.70:4047 , piece_id=ZNKZHAJAP24ZF6ZBV6762YY7LCRUSQCCL5E53WU5TMZ457CRZ3RA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.508011640 (282h21m34.059482111s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.897112773 (282h21m34.448583234s) [noise=true , address=81.171.14.70:4031 , piece_id=LYWET76RCJLTUN3DJDSVMDOZ6X5S53KPZ264P7D5FS7XCRDSYIQA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.027285292 (282h21m34.578755743s) [noise=true , address=81.171.14.70:4045 , piece_id=EOJUCJVNYJIF447J6FKIRVIXM3IJ5OU6TEVUKHFKQCUUVIMBR2EA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.070911284 (282h21m34.622381745s) [noise=true , address=89.149.201.147:4011 , piece_id=F5544PH4I4Z224P2DVMAXLM57DWIFT37KKXUJBR7B3OG5FV7KK6A, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.073062550 (282h21m34.624533011s) [noise=true , address=89.149.201.147:4018 , piece_id=LPC2AQ7TX4UUEI4I5VMGDOIFJI7R3GJGAOL5Q2O7SVY42OFN3XVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.083744591 (282h21m34.635215052s) [noise=true , address=89.149.201.147:4008 , piece_id=GSCXEJOMPWFECP7YSD66BRY26ZTXGIRZIMZYWKIW2SLRGPSJT7OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.146992305 (282h21m34.698462766s) [noise=true , address=89.149.201.147:4003 , piece_id=SDOIACESWFHPNPOZUZIXW5F2K62OUKCLCBHUKD4YKPLUQG4XXE5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.147018144 (282h21m34.698488605s) [seg_pos={0 0}, upload=248651]: Segment reached optimal threshold of 26 pieces. Nov 21 02:25:36.147392256 (282h21m34.698862717s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Nov 21 02:25:36.147403267 (282h21m34.698873728s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148140441 (282h21m34.699610933s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9): protocol: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection Nov 21 02:25:36.148153566 (282h21m34.699624027s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148213310 (282h21m34.699683761s) [seg_pos={0 0}, upload=248651]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 02:25:36.148357414 (282h21m34.699827875s) [upload=248651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 02:25:36.148395076 (282h21m34.699865547s) [upload=248651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 02:25:36.406626090 (282h21m34.958096551s) [upload=248651]: Done uploading segments... Nov 21 02:25:36.406651539 (282h21m34.958122s) [upload=248651]: Upload finished. err: <nil> Duration: 6.285341554s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.016479592s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:06:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.016508466s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............X.............X......X.............X..........] duration: 3.915910727s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:06:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.915930333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.36079016s trace: https://jaeger.dev.storj.io/trace/5860d9055b13d3d5 FAIL (2025-11-21 03:06:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02:25:33.629924468 (282h21m32.181394939s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630349578 (282h21m32.181820039s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630843158 (282h21m32.182313619s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.631501040 (282h21m32.182971502s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.632877661 (282h21m32.184348112s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.633957257 (282h21m32.185427718s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635008760 (282h21m32.186479221s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635936546 (282h21m32.187407007s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.636636048 (282h21m32.188106509s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:34.461042315 (282h21m33.012512776s) [upload=248651]: Next returned nil segment Nov 21 02:25:34.461079235 (282h21m33.012549697s) [upload=248651]: Waiting for error group managing segments... Nov 21 02:25:34.850359259 (282h21m33.40182972s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.852076179 (282h21m33.40354664s) [noise=true , address=185.167.204.206:4055 , piece_id=H54UI2MK43VZPELPTCU6XNLESYE6S5XZ4ZKAOV66TP3BBKKKDMFA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.856988012 (282h21m33.408458473s) [noise=true , address=185.167.204.205:4031 , piece_id=MCKJTCBSRINYPN27XNNFMXYPDHE3KVMSCWOB7VCOETNIO4F3PZVQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858141178 (282h21m33.409611639s) [noise=true , address=185.167.204.206:4067 , piece_id=TQD5JG4TGZYKCG4KOAWEUF6BSVY2IX35O2IB6VYKKH7ABNQEMRRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858405160 (282h21m33.409875621s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858612054 (282h21m33.410082515s) [noise=true , address=185.167.204.205:4023 , piece_id=NQMCNM7GAS2URY7JNN4DXMU2FZ7UAVHC42U55IVIGYUWXZ5DFOCQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859019680 (282h21m33.410490131s) [noise=true , address=185.167.204.205:4013 , piece_id=G5AFIVUYYIG3IAVQ2UN4MTL3NPXN326IOR77CXTU4BAIBFMICKXQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859464347 (282h21m33.410934808s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859565901 (282h21m33.411036352s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866136853 (282h21m33.417607324s) [noise=true , address=185.167.204.205:4027 , piece_id=DTF2OMEXXT2XFQB2RQOKL66K4AOIC3DTZOTE2ZGIMYWVPCTKJWJA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866678734 (282h21m33.418149195s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867348910 (282h21m33.418819361s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867982197 (282h21m33.419452668s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.868482399 (282h21m33.41995286s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.880010110 (282h21m33.431480601s) [noise=true , address=185.167.204.205:4025 , piece_id=ZWXX3V2QD4QSVDNMMDJ5GSSLIMSAK5IZOHOEB2SI3TFMPUXRRVXA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.419467556 (282h21m33.970938017s) [noise=true , address=81.171.14.70:4044 , piece_id=WAQ3SBHX6TRISFEOPE4Y6O2HSE2NYVC257V6ROMZSFVJO7YLCGRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.424133200 (282h21m33.975603661s) [noise=true , address=81.171.14.70:4038 , piece_id=X67ZAHX3AIYBBWES2PL5TB6FKR3DGZBALSLTITODERBWUI2OILVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.433560922 (282h21m33.985031383s) [noise=true , address=81.171.14.70:4041 , piece_id=AAUJYA6S7AWTCUO33QBNLCFUKRLGY6NLGFHRECJ5I63H53J46SMQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.452011215 (282h21m34.003481676s) [noise=true , address=81.171.14.70:4047 , piece_id=ZNKZHAJAP24ZF6ZBV6762YY7LCRUSQCCL5E53WU5TMZ457CRZ3RA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.508011640 (282h21m34.059482111s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.897112773 (282h21m34.448583234s) [noise=true , address=81.171.14.70:4031 , piece_id=LYWET76RCJLTUN3DJDSVMDOZ6X5S53KPZ264P7D5FS7XCRDSYIQA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.027285292 (282h21m34.578755743s) [noise=true , address=81.171.14.70:4045 , piece_id=EOJUCJVNYJIF447J6FKIRVIXM3IJ5OU6TEVUKHFKQCUUVIMBR2EA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.070911284 (282h21m34.622381745s) [noise=true , address=89.149.201.147:4011 , piece_id=F5544PH4I4Z224P2DVMAXLM57DWIFT37KKXUJBR7B3OG5FV7KK6A, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.073062550 (282h21m34.624533011s) [noise=true , address=89.149.201.147:4018 , piece_id=LPC2AQ7TX4UUEI4I5VMGDOIFJI7R3GJGAOL5Q2O7SVY42OFN3XVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.083744591 (282h21m34.635215052s) [noise=true , address=89.149.201.147:4008 , piece_id=GSCXEJOMPWFECP7YSD66BRY26ZTXGIRZIMZYWKIW2SLRGPSJT7OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.146992305 (282h21m34.698462766s) [noise=true , address=89.149.201.147:4003 , piece_id=SDOIACESWFHPNPOZUZIXW5F2K62OUKCLCBHUKD4YKPLUQG4XXE5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.147018144 (282h21m34.698488605s) [seg_pos={0 0}, upload=248651]: Segment reached optimal threshold of 26 pieces. Nov 21 02:25:36.147392256 (282h21m34.698862717s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Nov 21 02:25:36.147403267 (282h21m34.698873728s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148140441 (282h21m34.699610933s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9): protocol: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection Nov 21 02:25:36.148153566 (282h21m34.699624027s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148213310 (282h21m34.699683761s) [seg_pos={0 0}, upload=248651]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 02:25:36.148357414 (282h21m34.699827875s) [upload=248651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 02:25:36.148395076 (282h21m34.699865547s) [upload=248651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 02:25:36.406626090 (282h21m34.958096551s) [upload=248651]: Done uploading segments... Nov 21 02:25:36.406651539 (282h21m34.958122s) [upload=248651]: Upload finished. err: <nil> Duration: 5.360853751s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.776816147s trace: https://jaeger.dev.storj.io/trace/5860d9055b13d3d5 FAIL (2025-11-21 03:06:02 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02:25:33.629924468 (282h21m32.181394939s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630349578 (282h21m32.181820039s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630843158 (282h21m32.182313619s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.631501040 (282h21m32.182971502s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.632877661 (282h21m32.184348112s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.633957257 (282h21m32.185427718s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635008760 (282h21m32.186479221s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635936546 (282h21m32.187407007s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.636636048 (282h21m32.188106509s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:34.461042315 (282h21m33.012512776s) [upload=248651]: Next returned nil segment Nov 21 02:25:34.461079235 (282h21m33.012549697s) [upload=248651]: Waiting for error group managing segments... Nov 21 02:25:34.850359259 (282h21m33.40182972s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.852076179 (282h21m33.40354664s) [noise=true , address=185.167.204.206:4055 , piece_id=H54UI2MK43VZPELPTCU6XNLESYE6S5XZ4ZKAOV66TP3BBKKKDMFA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.856988012 (282h21m33.408458473s) [noise=true , address=185.167.204.205:4031 , piece_id=MCKJTCBSRINYPN27XNNFMXYPDHE3KVMSCWOB7VCOETNIO4F3PZVQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858141178 (282h21m33.409611639s) [noise=true , address=185.167.204.206:4067 , piece_id=TQD5JG4TGZYKCG4KOAWEUF6BSVY2IX35O2IB6VYKKH7ABNQEMRRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858405160 (282h21m33.409875621s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858612054 (282h21m33.410082515s) [noise=true , address=185.167.204.205:4023 , piece_id=NQMCNM7GAS2URY7JNN4DXMU2FZ7UAVHC42U55IVIGYUWXZ5DFOCQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859019680 (282h21m33.410490131s) [noise=true , address=185.167.204.205:4013 , piece_id=G5AFIVUYYIG3IAVQ2UN4MTL3NPXN326IOR77CXTU4BAIBFMICKXQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859464347 (282h21m33.410934808s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859565901 (282h21m33.411036352s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866136853 (282h21m33.417607324s) [noise=true , address=185.167.204.205:4027 , piece_id=DTF2OMEXXT2XFQB2RQOKL66K4AOIC3DTZOTE2ZGIMYWVPCTKJWJA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866678734 (282h21m33.418149195s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867348910 (282h21m33.418819361s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867982197 (282h21m33.419452668s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.868482399 (282h21m33.41995286s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.880010110 (282h21m33.431480601s) [noise=true , address=185.167.204.205:4025 , piece_id=ZWXX3V2QD4QSVDNMMDJ5GSSLIMSAK5IZOHOEB2SI3TFMPUXRRVXA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.419467556 (282h21m33.970938017s) [noise=true , address=81.171.14.70:4044 , piece_id=WAQ3SBHX6TRISFEOPE4Y6O2HSE2NYVC257V6ROMZSFVJO7YLCGRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.424133200 (282h21m33.975603661s) [noise=true , address=81.171.14.70:4038 , piece_id=X67ZAHX3AIYBBWES2PL5TB6FKR3DGZBALSLTITODERBWUI2OILVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.433560922 (282h21m33.985031383s) [noise=true , address=81.171.14.70:4041 , piece_id=AAUJYA6S7AWTCUO33QBNLCFUKRLGY6NLGFHRECJ5I63H53J46SMQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.452011215 (282h21m34.003481676s) [noise=true , address=81.171.14.70:4047 , piece_id=ZNKZHAJAP24ZF6ZBV6762YY7LCRUSQCCL5E53WU5TMZ457CRZ3RA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.508011640 (282h21m34.059482111s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.897112773 (282h21m34.448583234s) [noise=true , address=81.171.14.70:4031 , piece_id=LYWET76RCJLTUN3DJDSVMDOZ6X5S53KPZ264P7D5FS7XCRDSYIQA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.027285292 (282h21m34.578755743s) [noise=true , address=81.171.14.70:4045 , piece_id=EOJUCJVNYJIF447J6FKIRVIXM3IJ5OU6TEVUKHFKQCUUVIMBR2EA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.070911284 (282h21m34.622381745s) [noise=true , address=89.149.201.147:4011 , piece_id=F5544PH4I4Z224P2DVMAXLM57DWIFT37KKXUJBR7B3OG5FV7KK6A, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.073062550 (282h21m34.624533011s) [noise=true , address=89.149.201.147:4018 , piece_id=LPC2AQ7TX4UUEI4I5VMGDOIFJI7R3GJGAOL5Q2O7SVY42OFN3XVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.083744591 (282h21m34.635215052s) [noise=true , address=89.149.201.147:4008 , piece_id=GSCXEJOMPWFECP7YSD66BRY26ZTXGIRZIMZYWKIW2SLRGPSJT7OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.146992305 (282h21m34.698462766s) [noise=true , address=89.149.201.147:4003 , piece_id=SDOIACESWFHPNPOZUZIXW5F2K62OUKCLCBHUKD4YKPLUQG4XXE5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.147018144 (282h21m34.698488605s) [seg_pos={0 0}, upload=248651]: Segment reached optimal threshold of 26 pieces. Nov 21 02:25:36.147392256 (282h21m34.698862717s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Nov 21 02:25:36.147403267 (282h21m34.698873728s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148140441 (282h21m34.699610933s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9): protocol: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection Nov 21 02:25:36.148153566 (282h21m34.699624027s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148213310 (282h21m34.699683761s) [seg_pos={0 0}, upload=248651]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 02:25:36.148357414 (282h21m34.699827875s) [upload=248651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 02:25:36.148395076 (282h21m34.699865547s) [upload=248651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 02:25:36.406626090 (282h21m34.958096551s) [upload=248651]: Done uploading segments... Nov 21 02:25:36.406651539 (282h21m34.958122s) [upload=248651]: Upload finished. err: <nil> Duration: 5.776849252s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............X.............X......X.............X.........] duration: 4.953366232s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:06:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.953397251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [................X.............X......X.............X........] duration: 4.085428284s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:05:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.085549164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.745785209s trace: https://jaeger.dev.storj.io/trace/5860d9055b13d3d5 FAIL (2025-11-21 03:05:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02:25:33.629924468 (282h21m32.181394939s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630349578 (282h21m32.181820039s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630843158 (282h21m32.182313619s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.631501040 (282h21m32.182971502s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.632877661 (282h21m32.184348112s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.633957257 (282h21m32.185427718s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635008760 (282h21m32.186479221s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635936546 (282h21m32.187407007s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.636636048 (282h21m32.188106509s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:34.461042315 (282h21m33.012512776s) [upload=248651]: Next returned nil segment Nov 21 02:25:34.461079235 (282h21m33.012549697s) [upload=248651]: Waiting for error group managing segments... Nov 21 02:25:34.850359259 (282h21m33.40182972s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.852076179 (282h21m33.40354664s) [noise=true , address=185.167.204.206:4055 , piece_id=H54UI2MK43VZPELPTCU6XNLESYE6S5XZ4ZKAOV66TP3BBKKKDMFA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.856988012 (282h21m33.408458473s) [noise=true , address=185.167.204.205:4031 , piece_id=MCKJTCBSRINYPN27XNNFMXYPDHE3KVMSCWOB7VCOETNIO4F3PZVQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858141178 (282h21m33.409611639s) [noise=true , address=185.167.204.206:4067 , piece_id=TQD5JG4TGZYKCG4KOAWEUF6BSVY2IX35O2IB6VYKKH7ABNQEMRRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858405160 (282h21m33.409875621s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858612054 (282h21m33.410082515s) [noise=true , address=185.167.204.205:4023 , piece_id=NQMCNM7GAS2URY7JNN4DXMU2FZ7UAVHC42U55IVIGYUWXZ5DFOCQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859019680 (282h21m33.410490131s) [noise=true , address=185.167.204.205:4013 , piece_id=G5AFIVUYYIG3IAVQ2UN4MTL3NPXN326IOR77CXTU4BAIBFMICKXQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859464347 (282h21m33.410934808s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859565901 (282h21m33.411036352s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866136853 (282h21m33.417607324s) [noise=true , address=185.167.204.205:4027 , piece_id=DTF2OMEXXT2XFQB2RQOKL66K4AOIC3DTZOTE2ZGIMYWVPCTKJWJA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866678734 (282h21m33.418149195s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867348910 (282h21m33.418819361s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867982197 (282h21m33.419452668s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.868482399 (282h21m33.41995286s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.880010110 (282h21m33.431480601s) [noise=true , address=185.167.204.205:4025 , piece_id=ZWXX3V2QD4QSVDNMMDJ5GSSLIMSAK5IZOHOEB2SI3TFMPUXRRVXA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.419467556 (282h21m33.970938017s) [noise=true , address=81.171.14.70:4044 , piece_id=WAQ3SBHX6TRISFEOPE4Y6O2HSE2NYVC257V6ROMZSFVJO7YLCGRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.424133200 (282h21m33.975603661s) [noise=true , address=81.171.14.70:4038 , piece_id=X67ZAHX3AIYBBWES2PL5TB6FKR3DGZBALSLTITODERBWUI2OILVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.433560922 (282h21m33.985031383s) [noise=true , address=81.171.14.70:4041 , piece_id=AAUJYA6S7AWTCUO33QBNLCFUKRLGY6NLGFHRECJ5I63H53J46SMQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.452011215 (282h21m34.003481676s) [noise=true , address=81.171.14.70:4047 , piece_id=ZNKZHAJAP24ZF6ZBV6762YY7LCRUSQCCL5E53WU5TMZ457CRZ3RA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.508011640 (282h21m34.059482111s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.897112773 (282h21m34.448583234s) [noise=true , address=81.171.14.70:4031 , piece_id=LYWET76RCJLTUN3DJDSVMDOZ6X5S53KPZ264P7D5FS7XCRDSYIQA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.027285292 (282h21m34.578755743s) [noise=true , address=81.171.14.70:4045 , piece_id=EOJUCJVNYJIF447J6FKIRVIXM3IJ5OU6TEVUKHFKQCUUVIMBR2EA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.070911284 (282h21m34.622381745s) [noise=true , address=89.149.201.147:4011 , piece_id=F5544PH4I4Z224P2DVMAXLM57DWIFT37KKXUJBR7B3OG5FV7KK6A, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.073062550 (282h21m34.624533011s) [noise=true , address=89.149.201.147:4018 , piece_id=LPC2AQ7TX4UUEI4I5VMGDOIFJI7R3GJGAOL5Q2O7SVY42OFN3XVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.083744591 (282h21m34.635215052s) [noise=true , address=89.149.201.147:4008 , piece_id=GSCXEJOMPWFECP7YSD66BRY26ZTXGIRZIMZYWKIW2SLRGPSJT7OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.146992305 (282h21m34.698462766s) [noise=true , address=89.149.201.147:4003 , piece_id=SDOIACESWFHPNPOZUZIXW5F2K62OUKCLCBHUKD4YKPLUQG4XXE5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.147018144 (282h21m34.698488605s) [seg_pos={0 0}, upload=248651]: Segment reached optimal threshold of 26 pieces. Nov 21 02:25:36.147392256 (282h21m34.698862717s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Nov 21 02:25:36.147403267 (282h21m34.698873728s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148140441 (282h21m34.699610933s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9): protocol: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection Nov 21 02:25:36.148153566 (282h21m34.699624027s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148213310 (282h21m34.699683761s) [seg_pos={0 0}, upload=248651]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 02:25:36.148357414 (282h21m34.699827875s) [upload=248651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 02:25:36.148395076 (282h21m34.699865547s) [upload=248651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 02:25:36.406626090 (282h21m34.958096551s) [upload=248651]: Done uploading segments... Nov 21 02:25:36.406651539 (282h21m34.958122s) [upload=248651]: Upload finished. err: <nil> Duration: 5.74591113s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.98756117s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:05:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.987610463s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.892299626s trace: https://jaeger.dev.storj.io/trace/5860d9055b13d3d5 FAIL (2025-11-21 03:04:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02:25:33.629924468 (282h21m32.181394939s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630349578 (282h21m32.181820039s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630843158 (282h21m32.182313619s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.631501040 (282h21m32.182971502s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.632877661 (282h21m32.184348112s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.633957257 (282h21m32.185427718s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635008760 (282h21m32.186479221s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635936546 (282h21m32.187407007s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.636636048 (282h21m32.188106509s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:34.461042315 (282h21m33.012512776s) [upload=248651]: Next returned nil segment Nov 21 02:25:34.461079235 (282h21m33.012549697s) [upload=248651]: Waiting for error group managing segments... Nov 21 02:25:34.850359259 (282h21m33.40182972s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.852076179 (282h21m33.40354664s) [noise=true , address=185.167.204.206:4055 , piece_id=H54UI2MK43VZPELPTCU6XNLESYE6S5XZ4ZKAOV66TP3BBKKKDMFA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.856988012 (282h21m33.408458473s) [noise=true , address=185.167.204.205:4031 , piece_id=MCKJTCBSRINYPN27XNNFMXYPDHE3KVMSCWOB7VCOETNIO4F3PZVQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858141178 (282h21m33.409611639s) [noise=true , address=185.167.204.206:4067 , piece_id=TQD5JG4TGZYKCG4KOAWEUF6BSVY2IX35O2IB6VYKKH7ABNQEMRRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858405160 (282h21m33.409875621s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858612054 (282h21m33.410082515s) [noise=true , address=185.167.204.205:4023 , piece_id=NQMCNM7GAS2URY7JNN4DXMU2FZ7UAVHC42U55IVIGYUWXZ5DFOCQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859019680 (282h21m33.410490131s) [noise=true , address=185.167.204.205:4013 , piece_id=G5AFIVUYYIG3IAVQ2UN4MTL3NPXN326IOR77CXTU4BAIBFMICKXQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859464347 (282h21m33.410934808s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859565901 (282h21m33.411036352s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866136853 (282h21m33.417607324s) [noise=true , address=185.167.204.205:4027 , piece_id=DTF2OMEXXT2XFQB2RQOKL66K4AOIC3DTZOTE2ZGIMYWVPCTKJWJA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866678734 (282h21m33.418149195s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867348910 (282h21m33.418819361s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867982197 (282h21m33.419452668s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.868482399 (282h21m33.41995286s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.880010110 (282h21m33.431480601s) [noise=true , address=185.167.204.205:4025 , piece_id=ZWXX3V2QD4QSVDNMMDJ5GSSLIMSAK5IZOHOEB2SI3TFMPUXRRVXA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.419467556 (282h21m33.970938017s) [noise=true , address=81.171.14.70:4044 , piece_id=WAQ3SBHX6TRISFEOPE4Y6O2HSE2NYVC257V6ROMZSFVJO7YLCGRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.424133200 (282h21m33.975603661s) [noise=true , address=81.171.14.70:4038 , piece_id=X67ZAHX3AIYBBWES2PL5TB6FKR3DGZBALSLTITODERBWUI2OILVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.433560922 (282h21m33.985031383s) [noise=true , address=81.171.14.70:4041 , piece_id=AAUJYA6S7AWTCUO33QBNLCFUKRLGY6NLGFHRECJ5I63H53J46SMQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.452011215 (282h21m34.003481676s) [noise=true , address=81.171.14.70:4047 , piece_id=ZNKZHAJAP24ZF6ZBV6762YY7LCRUSQCCL5E53WU5TMZ457CRZ3RA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.508011640 (282h21m34.059482111s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.897112773 (282h21m34.448583234s) [noise=true , address=81.171.14.70:4031 , piece_id=LYWET76RCJLTUN3DJDSVMDOZ6X5S53KPZ264P7D5FS7XCRDSYIQA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.027285292 (282h21m34.578755743s) [noise=true , address=81.171.14.70:4045 , piece_id=EOJUCJVNYJIF447J6FKIRVIXM3IJ5OU6TEVUKHFKQCUUVIMBR2EA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.070911284 (282h21m34.622381745s) [noise=true , address=89.149.201.147:4011 , piece_id=F5544PH4I4Z224P2DVMAXLM57DWIFT37KKXUJBR7B3OG5FV7KK6A, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.073062550 (282h21m34.624533011s) [noise=true , address=89.149.201.147:4018 , piece_id=LPC2AQ7TX4UUEI4I5VMGDOIFJI7R3GJGAOL5Q2O7SVY42OFN3XVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.083744591 (282h21m34.635215052s) [noise=true , address=89.149.201.147:4008 , piece_id=GSCXEJOMPWFECP7YSD66BRY26ZTXGIRZIMZYWKIW2SLRGPSJT7OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.146992305 (282h21m34.698462766s) [noise=true , address=89.149.201.147:4003 , piece_id=SDOIACESWFHPNPOZUZIXW5F2K62OUKCLCBHUKD4YKPLUQG4XXE5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.147018144 (282h21m34.698488605s) [seg_pos={0 0}, upload=248651]: Segment reached optimal threshold of 26 pieces. Nov 21 02:25:36.147392256 (282h21m34.698862717s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Nov 21 02:25:36.147403267 (282h21m34.698873728s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148140441 (282h21m34.699610933s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9): protocol: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection Nov 21 02:25:36.148153566 (282h21m34.699624027s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148213310 (282h21m34.699683761s) [seg_pos={0 0}, upload=248651]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 02:25:36.148357414 (282h21m34.699827875s) [upload=248651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 02:25:36.148395076 (282h21m34.699865547s) [upload=248651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 02:25:36.406626090 (282h21m34.958096551s) [upload=248651]: Done uploading segments... Nov 21 02:25:36.406651539 (282h21m34.958122s) [upload=248651]: Upload finished. err: <nil> Duration: 5.89233367s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.................X.............X......X.............X.......] duration: 4.14447617s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:04:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.144503702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.878846939s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:04:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.878905361s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.246825178s trace: https://jaeger.dev.storj.io/trace/5860d9055b13d3d5 FAIL (2025-11-21 03:03:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02:25:33.629924468 (282h21m32.181394939s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630349578 (282h21m32.181820039s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630843158 (282h21m32.182313619s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.631501040 (282h21m32.182971502s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.632877661 (282h21m32.184348112s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.633957257 (282h21m32.185427718s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635008760 (282h21m32.186479221s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635936546 (282h21m32.187407007s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.636636048 (282h21m32.188106509s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:34.461042315 (282h21m33.012512776s) [upload=248651]: Next returned nil segment Nov 21 02:25:34.461079235 (282h21m33.012549697s) [upload=248651]: Waiting for error group managing segments... Nov 21 02:25:34.850359259 (282h21m33.40182972s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.852076179 (282h21m33.40354664s) [noise=true , address=185.167.204.206:4055 , piece_id=H54UI2MK43VZPELPTCU6XNLESYE6S5XZ4ZKAOV66TP3BBKKKDMFA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.856988012 (282h21m33.408458473s) [noise=true , address=185.167.204.205:4031 , piece_id=MCKJTCBSRINYPN27XNNFMXYPDHE3KVMSCWOB7VCOETNIO4F3PZVQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858141178 (282h21m33.409611639s) [noise=true , address=185.167.204.206:4067 , piece_id=TQD5JG4TGZYKCG4KOAWEUF6BSVY2IX35O2IB6VYKKH7ABNQEMRRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858405160 (282h21m33.409875621s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858612054 (282h21m33.410082515s) [noise=true , address=185.167.204.205:4023 , piece_id=NQMCNM7GAS2URY7JNN4DXMU2FZ7UAVHC42U55IVIGYUWXZ5DFOCQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859019680 (282h21m33.410490131s) [noise=true , address=185.167.204.205:4013 , piece_id=G5AFIVUYYIG3IAVQ2UN4MTL3NPXN326IOR77CXTU4BAIBFMICKXQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859464347 (282h21m33.410934808s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859565901 (282h21m33.411036352s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866136853 (282h21m33.417607324s) [noise=true , address=185.167.204.205:4027 , piece_id=DTF2OMEXXT2XFQB2RQOKL66K4AOIC3DTZOTE2ZGIMYWVPCTKJWJA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866678734 (282h21m33.418149195s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867348910 (282h21m33.418819361s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867982197 (282h21m33.419452668s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.868482399 (282h21m33.41995286s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.880010110 (282h21m33.431480601s) [noise=true , address=185.167.204.205:4025 , piece_id=ZWXX3V2QD4QSVDNMMDJ5GSSLIMSAK5IZOHOEB2SI3TFMPUXRRVXA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.419467556 (282h21m33.970938017s) [noise=true , address=81.171.14.70:4044 , piece_id=WAQ3SBHX6TRISFEOPE4Y6O2HSE2NYVC257V6ROMZSFVJO7YLCGRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.424133200 (282h21m33.975603661s) [noise=true , address=81.171.14.70:4038 , piece_id=X67ZAHX3AIYBBWES2PL5TB6FKR3DGZBALSLTITODERBWUI2OILVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.433560922 (282h21m33.985031383s) [noise=true , address=81.171.14.70:4041 , piece_id=AAUJYA6S7AWTCUO33QBNLCFUKRLGY6NLGFHRECJ5I63H53J46SMQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.452011215 (282h21m34.003481676s) [noise=true , address=81.171.14.70:4047 , piece_id=ZNKZHAJAP24ZF6ZBV6762YY7LCRUSQCCL5E53WU5TMZ457CRZ3RA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.508011640 (282h21m34.059482111s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.897112773 (282h21m34.448583234s) [noise=true , address=81.171.14.70:4031 , piece_id=LYWET76RCJLTUN3DJDSVMDOZ6X5S53KPZ264P7D5FS7XCRDSYIQA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.027285292 (282h21m34.578755743s) [noise=true , address=81.171.14.70:4045 , piece_id=EOJUCJVNYJIF447J6FKIRVIXM3IJ5OU6TEVUKHFKQCUUVIMBR2EA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.070911284 (282h21m34.622381745s) [noise=true , address=89.149.201.147:4011 , piece_id=F5544PH4I4Z224P2DVMAXLM57DWIFT37KKXUJBR7B3OG5FV7KK6A, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.073062550 (282h21m34.624533011s) [noise=true , address=89.149.201.147:4018 , piece_id=LPC2AQ7TX4UUEI4I5VMGDOIFJI7R3GJGAOL5Q2O7SVY42OFN3XVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.083744591 (282h21m34.635215052s) [noise=true , address=89.149.201.147:4008 , piece_id=GSCXEJOMPWFECP7YSD66BRY26ZTXGIRZIMZYWKIW2SLRGPSJT7OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.146992305 (282h21m34.698462766s) [noise=true , address=89.149.201.147:4003 , piece_id=SDOIACESWFHPNPOZUZIXW5F2K62OUKCLCBHUKD4YKPLUQG4XXE5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.147018144 (282h21m34.698488605s) [seg_pos={0 0}, upload=248651]: Segment reached optimal threshold of 26 pieces. Nov 21 02:25:36.147392256 (282h21m34.698862717s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Nov 21 02:25:36.147403267 (282h21m34.698873728s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148140441 (282h21m34.699610933s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9): protocol: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection Nov 21 02:25:36.148153566 (282h21m34.699624027s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148213310 (282h21m34.699683761s) [seg_pos={0 0}, upload=248651]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 02:25:36.148357414 (282h21m34.699827875s) [upload=248651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 02:25:36.148395076 (282h21m34.699865547s) [upload=248651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 02:25:36.406626090 (282h21m34.958096551s) [upload=248651]: Done uploading segments... Nov 21 02:25:36.406651539 (282h21m34.958122s) [upload=248651]: Upload finished. err: <nil> Duration: 6.246851727s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..................X.............X......X.............X......] duration: 4.123813255s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:03:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.123842519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.836826167s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:03:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.836853648s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...................X.............X......X.............X.....] duration: 4.123571084s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:03:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.123590762s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....................X.............X......X.............X....] duration: 5.000997533s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:02:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00108015s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.723259063s trace: https://jaeger.dev.storj.io/trace/5860d9055b13d3d5 FAIL (2025-11-21 03:02:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02:25:33.629924468 (282h21m32.181394939s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630349578 (282h21m32.181820039s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630843158 (282h21m32.182313619s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.631501040 (282h21m32.182971502s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.632877661 (282h21m32.184348112s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.633957257 (282h21m32.185427718s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635008760 (282h21m32.186479221s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635936546 (282h21m32.187407007s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.636636048 (282h21m32.188106509s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:34.461042315 (282h21m33.012512776s) [upload=248651]: Next returned nil segment Nov 21 02:25:34.461079235 (282h21m33.012549697s) [upload=248651]: Waiting for error group managing segments... Nov 21 02:25:34.850359259 (282h21m33.40182972s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.852076179 (282h21m33.40354664s) [noise=true , address=185.167.204.206:4055 , piece_id=H54UI2MK43VZPELPTCU6XNLESYE6S5XZ4ZKAOV66TP3BBKKKDMFA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.856988012 (282h21m33.408458473s) [noise=true , address=185.167.204.205:4031 , piece_id=MCKJTCBSRINYPN27XNNFMXYPDHE3KVMSCWOB7VCOETNIO4F3PZVQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858141178 (282h21m33.409611639s) [noise=true , address=185.167.204.206:4067 , piece_id=TQD5JG4TGZYKCG4KOAWEUF6BSVY2IX35O2IB6VYKKH7ABNQEMRRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858405160 (282h21m33.409875621s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858612054 (282h21m33.410082515s) [noise=true , address=185.167.204.205:4023 , piece_id=NQMCNM7GAS2URY7JNN4DXMU2FZ7UAVHC42U55IVIGYUWXZ5DFOCQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859019680 (282h21m33.410490131s) [noise=true , address=185.167.204.205:4013 , piece_id=G5AFIVUYYIG3IAVQ2UN4MTL3NPXN326IOR77CXTU4BAIBFMICKXQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859464347 (282h21m33.410934808s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859565901 (282h21m33.411036352s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866136853 (282h21m33.417607324s) [noise=true , address=185.167.204.205:4027 , piece_id=DTF2OMEXXT2XFQB2RQOKL66K4AOIC3DTZOTE2ZGIMYWVPCTKJWJA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866678734 (282h21m33.418149195s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867348910 (282h21m33.418819361s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867982197 (282h21m33.419452668s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.868482399 (282h21m33.41995286s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.880010110 (282h21m33.431480601s) [noise=true , address=185.167.204.205:4025 , piece_id=ZWXX3V2QD4QSVDNMMDJ5GSSLIMSAK5IZOHOEB2SI3TFMPUXRRVXA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.419467556 (282h21m33.970938017s) [noise=true , address=81.171.14.70:4044 , piece_id=WAQ3SBHX6TRISFEOPE4Y6O2HSE2NYVC257V6ROMZSFVJO7YLCGRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.424133200 (282h21m33.975603661s) [noise=true , address=81.171.14.70:4038 , piece_id=X67ZAHX3AIYBBWES2PL5TB6FKR3DGZBALSLTITODERBWUI2OILVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.433560922 (282h21m33.985031383s) [noise=true , address=81.171.14.70:4041 , piece_id=AAUJYA6S7AWTCUO33QBNLCFUKRLGY6NLGFHRECJ5I63H53J46SMQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.452011215 (282h21m34.003481676s) [noise=true , address=81.171.14.70:4047 , piece_id=ZNKZHAJAP24ZF6ZBV6762YY7LCRUSQCCL5E53WU5TMZ457CRZ3RA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.508011640 (282h21m34.059482111s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.897112773 (282h21m34.448583234s) [noise=true , address=81.171.14.70:4031 , piece_id=LYWET76RCJLTUN3DJDSVMDOZ6X5S53KPZ264P7D5FS7XCRDSYIQA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.027285292 (282h21m34.578755743s) [noise=true , address=81.171.14.70:4045 , piece_id=EOJUCJVNYJIF447J6FKIRVIXM3IJ5OU6TEVUKHFKQCUUVIMBR2EA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.070911284 (282h21m34.622381745s) [noise=true , address=89.149.201.147:4011 , piece_id=F5544PH4I4Z224P2DVMAXLM57DWIFT37KKXUJBR7B3OG5FV7KK6A, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.073062550 (282h21m34.624533011s) [noise=true , address=89.149.201.147:4018 , piece_id=LPC2AQ7TX4UUEI4I5VMGDOIFJI7R3GJGAOL5Q2O7SVY42OFN3XVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.083744591 (282h21m34.635215052s) [noise=true , address=89.149.201.147:4008 , piece_id=GSCXEJOMPWFECP7YSD66BRY26ZTXGIRZIMZYWKIW2SLRGPSJT7OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.146992305 (282h21m34.698462766s) [noise=true , address=89.149.201.147:4003 , piece_id=SDOIACESWFHPNPOZUZIXW5F2K62OUKCLCBHUKD4YKPLUQG4XXE5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.147018144 (282h21m34.698488605s) [seg_pos={0 0}, upload=248651]: Segment reached optimal threshold of 26 pieces. Nov 21 02:25:36.147392256 (282h21m34.698862717s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Nov 21 02:25:36.147403267 (282h21m34.698873728s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148140441 (282h21m34.699610933s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9): protocol: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection Nov 21 02:25:36.148153566 (282h21m34.699624027s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148213310 (282h21m34.699683761s) [seg_pos={0 0}, upload=248651]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 02:25:36.148357414 (282h21m34.699827875s) [upload=248651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 02:25:36.148395076 (282h21m34.699865547s) [upload=248651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 02:25:36.406626090 (282h21m34.958096551s) [upload=248651]: Done uploading segments... Nov 21 02:25:36.406651539 (282h21m34.958122s) [upload=248651]: Upload finished. err: <nil> Duration: 5.723335239s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.864867369s trace: https://jaeger.dev.storj.io/trace/30b3234a6717a51a FAIL (2025-11-21 03:02:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 21 02:54:15.843592266 (282h50m14.395062727s) [upload=249071]: Starting upload Nov 21 02:54:15.843672789 (282h50m14.39514327s) [upload=249071]: Done starting upload Nov 21 02:54:15.843785233 (282h50m14.395255704s) [upload=249071]: Uploading segments... Nov 21 02:54:15.843934156 (282h50m14.395404617s) [upload=249071]: Got next segment. Inline: false Nov 21 02:54:15.843978902 (282h50m14.395449363s) [upload=249071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 02:54:15.843996795 (282h50m14.395467257s) [upload=249071]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 02:54:16.091950402 (282h50m14.643420863s) [seg_pos={0 0}, upload=249071]: Begin upload segment... Nov 21 02:54:16.092018621 (282h50m14.643489072s) [seg_pos={0 0}, upload=249071]: Done begin upload segment. Nov 21 02:54:16.092070510 (282h50m14.643540971s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092200388 (282h50m14.643670849s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.092858641 (282h50m14.644329112s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.093851992 (282h50m14.645322453s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094216958 (282h50m14.645687419s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.094588414 (282h50m14.646058876s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.095819129 (282h50m14.64728959s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.096419892 (282h50m14.647890353s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097236206 (282h50m14.648706657s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.097868211 (282h50m14.649338682s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098320622 (282h50m14.649791073s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.098783513 (282h50m14.650253974s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099304064 (282h50m14.650774526s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.099408423 (282h50m14.650878884s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100171877 (282h50m14.651642338s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:16.100479834 (282h50m14.651950295s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Uploading piece... Nov 21 02:54:17.932068471 (282h50m16.483538932s) [upload=249071]: Next returned nil segment Nov 21 02:54:17.932084962 (282h50m16.483555423s) [upload=249071]: Waiting for error group managing segments... Nov 21 02:54:18.224231288 (282h50m16.77570176s) [noise=true , address=45.91.116.11:4117 , piece_id=PZKSOZQOZROZMLPGEWGMXVDTI2WM6TTDS3DX3BJH52HZFPIOES7Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.274684561 (282h50m16.826155022s) [noise=true , address=45.91.116.11:4118 , piece_id=SW3QYNV6MSP4QBG5RZ54LXE3EEO2VHZDXYBPLKS5S34LDY35PXNA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.280784997 (282h50m16.832255458s) [noise=true , address=77.90.31.15:4698 , piece_id=6M3YRZIE2SWNLSRAQTLWNWLGCJCI4UKYFBY2FSU45XPMTIKEMGVQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.283496200 (282h50m16.834966651s) [noise=true , address=77.90.31.13:4627 , piece_id=INNCGJ5EDGRX73SDFS66RAMQEO5KHL4PMP4LHH4MUHIHJSCUEWGA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.286515007 (282h50m16.837985468s) [noise=true , address=77.90.31.19:4841 , piece_id=5OSNUW6NPFIYGPRKN2JMSTQDW43ZIC5MTCQFIUI45QWGXLB4MM6Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:18.287276277 (282h50m16.838746738s) [noise=true , address=45.91.116.7:4009 , piece_id=JPWHCYIWLESTZKQ7J5IVQ3XOPA4YHXDQQBGMJ3GVMGHDBXP4U24A, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.241168371 (282h50m17.792638832s) [noise=true , address=145.79.227.30:4761 , piece_id=UVBF3RK5CCY6WDDX4D5YLLQ7ZEZTQWFVXQKHMDID5M75TSZC35RQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.651749998 (282h50m18.20322045s) [noise=true , address=145.79.227.27:4691 , piece_id=4QWKSUE3PEEOVIE5A2SHP647VWVFGEHCBRAYM5P2D3TF6W5YMQKQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.861685411 (282h50m18.413155862s) [noise=true , address=145.79.227.15:4875 , piece_id=HLEE6S6GK3N5M62HXWUEV7HBZ25TFTOWZDMJJREGMKD2N7YC2ZIQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:19.891504472 (282h50m18.442974933s) [noise=true , address=145.79.227.15:4874 , piece_id=RZOMRFUM4NAYWX6ZJWYZ55NEQ7L2SL4HXM7MNUIXVWPWKVBAOIDA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:20.823700993 (282h50m19.375171444s) [noise=true , address=145.79.227.11:4729 , piece_id=UVEPHOMV5VUJ534CORKQUPLDM5HS2MUCCCUBG7SZ2Y3MGN7VFG2Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:21.526158773 (282h50m20.077629234s) [noise=true , address=145.79.227.24:4583 , piece_id=VOVKTSABSL2VVP53KHW7YEXM2A35N6CX3SVHEQFCSJFPE5IA55KQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: <nil> Nov 21 02:54:45.150216895 (282h50m43.701687356s) [noise=true , address=145.79.227.14:4837 , piece_id=SXCA376PRU6IJRXRMSZDR3F3FMEMZTDQC7RV2HDNWPRSC5UQRQFA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: context deadline exceeded; context deadline exceeded Nov 21 02:54:45.150273533 (282h50m43.701744004s) [noise=true , address=145.79.227.31:4798 , piece_id=VO23VOQTYN7Q67PLQRGIX5I7TGGQSEFRL7F6JHEAMISAGQX5A22Q, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->145.79.227.31:4798: use of closed network connection Nov 21 02:54:45.150496998 (282h50m43.701967459s) [noise=true , address=145.79.227.25:4617 , piece_id=V5NDFI6TVGC7N5UYVIOPUH32NIVBZ5YR3BRNNHA7UKTUTEWFVSOA, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34764->145.79.227.25:4617: use of closed network connection Nov 21 02:54:45.150988906 (282h50m43.702459367s) [noise=true , address=145.79.227.28:4727 , piece_id=CFMU7NLXMIIM6N3LGAZAJFGHYYUAQKA7IC67T73JEXMKNHLDBCXQ, seg_pos={0 0}, upload=249071]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: read tcp 188.166.225.170:46342->145.79.227.28:4727: use of closed network connection; context deadline exceeded Nov 21 02:54:45.151065552 (282h50m43.702536013s) [seg_pos={0 0}, upload=249071]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 21 02:54:45.393960414 (282h50m43.945430865s) [upload=249071]: Done uploading segments... Nov 21 02:54:45.393998446 (282h50m43.945468907s) [upload=249071]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.864889191s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.267829209s trace: https://jaeger.dev.storj.io/trace/5860d9055b13d3d5 FAIL (2025-11-21 03:02:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 02:25:33.629924468 (282h21m32.181394939s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630349578 (282h21m32.181820039s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.630843158 (282h21m32.182313619s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.631501040 (282h21m32.182971502s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.632877661 (282h21m32.184348112s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.633957257 (282h21m32.185427718s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635008760 (282h21m32.186479221s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.635936546 (282h21m32.187407007s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:33.636636048 (282h21m32.188106509s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Uploading piece... Nov 21 02:25:34.461042315 (282h21m33.012512776s) [upload=248651]: Next returned nil segment Nov 21 02:25:34.461079235 (282h21m33.012549697s) [upload=248651]: Waiting for error group managing segments... Nov 21 02:25:34.850359259 (282h21m33.40182972s) [noise=true , address=185.167.204.206:4056 , piece_id=WP4BU4B4BJKASBDB7R7CB4IVBIP7NVRIHAOTZ2LWPG3XHMMC65TA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.852076179 (282h21m33.40354664s) [noise=true , address=185.167.204.206:4055 , piece_id=H54UI2MK43VZPELPTCU6XNLESYE6S5XZ4ZKAOV66TP3BBKKKDMFA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.856988012 (282h21m33.408458473s) [noise=true , address=185.167.204.205:4031 , piece_id=MCKJTCBSRINYPN27XNNFMXYPDHE3KVMSCWOB7VCOETNIO4F3PZVQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858141178 (282h21m33.409611639s) [noise=true , address=185.167.204.206:4067 , piece_id=TQD5JG4TGZYKCG4KOAWEUF6BSVY2IX35O2IB6VYKKH7ABNQEMRRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858405160 (282h21m33.409875621s) [noise=true , address=185.167.204.206:4041 , piece_id=S7UUW2R74ETDDNDVA73DRMGZ65YTPPLPWILZZ626SIG2YG55AOOA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.858612054 (282h21m33.410082515s) [noise=true , address=185.167.204.205:4023 , piece_id=NQMCNM7GAS2URY7JNN4DXMU2FZ7UAVHC42U55IVIGYUWXZ5DFOCQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859019680 (282h21m33.410490131s) [noise=true , address=185.167.204.205:4013 , piece_id=G5AFIVUYYIG3IAVQ2UN4MTL3NPXN326IOR77CXTU4BAIBFMICKXQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859464347 (282h21m33.410934808s) [noise=true , address=185.167.204.206:4044 , piece_id=6TMH2M7X2J6NUJ7H7YQ44MK3CHAHTONHFVB767OLQ7LVWYS6SWSQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.859565901 (282h21m33.411036352s) [noise=true , address=185.167.204.206:4071 , piece_id=CWYOC5U5SIHG24FJA76NVMYJG47L55GDKXTZ7FHENXXGL6NNLNCA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866136853 (282h21m33.417607324s) [noise=true , address=185.167.204.205:4027 , piece_id=DTF2OMEXXT2XFQB2RQOKL66K4AOIC3DTZOTE2ZGIMYWVPCTKJWJA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.866678734 (282h21m33.418149195s) [noise=true , address=185.167.204.206:4043 , piece_id=OBBQPYJ5QWYI3IDE7ROOB62NCIA425S2RW6N4BRRNXDSN7XTH3VA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867348910 (282h21m33.418819361s) [noise=true , address=185.167.204.205:4015 , piece_id=UC43M7E3DTT7P6X5FZR63SXMW3B7Y7LNX3CIXI5YGWVIPVXHRWWQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.867982197 (282h21m33.419452668s) [noise=true , address=185.167.204.206:4058 , piece_id=WJFVIB3443AETF6JXCW2CAYSL2E5SZWD3XITVJICPRBJUU7BO2OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.868482399 (282h21m33.41995286s) [noise=true , address=185.167.204.206:4037 , piece_id=ZTVJJC7I2VQ3ZO62Y2SNZKV64WYLEXHL2CJNDORXID6NBWWZSOYQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:34.880010110 (282h21m33.431480601s) [noise=true , address=185.167.204.205:4025 , piece_id=ZWXX3V2QD4QSVDNMMDJ5GSSLIMSAK5IZOHOEB2SI3TFMPUXRRVXA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.419467556 (282h21m33.970938017s) [noise=true , address=81.171.14.70:4044 , piece_id=WAQ3SBHX6TRISFEOPE4Y6O2HSE2NYVC257V6ROMZSFVJO7YLCGRQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.424133200 (282h21m33.975603661s) [noise=true , address=81.171.14.70:4038 , piece_id=X67ZAHX3AIYBBWES2PL5TB6FKR3DGZBALSLTITODERBWUI2OILVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.433560922 (282h21m33.985031383s) [noise=true , address=81.171.14.70:4041 , piece_id=AAUJYA6S7AWTCUO33QBNLCFUKRLGY6NLGFHRECJ5I63H53J46SMQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.452011215 (282h21m34.003481676s) [noise=true , address=81.171.14.70:4047 , piece_id=ZNKZHAJAP24ZF6ZBV6762YY7LCRUSQCCL5E53WU5TMZ457CRZ3RA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.508011640 (282h21m34.059482111s) [noise=true , address=81.171.14.70:4035 , piece_id=VI2R7YZTWKO4NNTYCTKDPAMHAA2CA5UA54ZJJOQXJ5FG6YY4VS5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:35.897112773 (282h21m34.448583234s) [noise=true , address=81.171.14.70:4031 , piece_id=LYWET76RCJLTUN3DJDSVMDOZ6X5S53KPZ264P7D5FS7XCRDSYIQA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.027285292 (282h21m34.578755743s) [noise=true , address=81.171.14.70:4045 , piece_id=EOJUCJVNYJIF447J6FKIRVIXM3IJ5OU6TEVUKHFKQCUUVIMBR2EA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.070911284 (282h21m34.622381745s) [noise=true , address=89.149.201.147:4011 , piece_id=F5544PH4I4Z224P2DVMAXLM57DWIFT37KKXUJBR7B3OG5FV7KK6A, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.073062550 (282h21m34.624533011s) [noise=true , address=89.149.201.147:4018 , piece_id=LPC2AQ7TX4UUEI4I5VMGDOIFJI7R3GJGAOL5Q2O7SVY42OFN3XVA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.083744591 (282h21m34.635215052s) [noise=true , address=89.149.201.147:4008 , piece_id=GSCXEJOMPWFECP7YSD66BRY26ZTXGIRZIMZYWKIW2SLRGPSJT7OQ, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.146992305 (282h21m34.698462766s) [noise=true , address=89.149.201.147:4003 , piece_id=SDOIACESWFHPNPOZUZIXW5F2K62OUKCLCBHUKD4YKPLUQG4XXE5Q, seg_pos={0 0}, upload=248651]: Done uploading piece. err: <nil> Nov 21 02:25:36.147018144 (282h21m34.698488605s) [seg_pos={0 0}, upload=248651]: Segment reached optimal threshold of 26 pieces. Nov 21 02:25:36.147392256 (282h21m34.698862717s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Nov 21 02:25:36.147403267 (282h21m34.698873728s) [noise=true , address=89.149.201.147:4021 , piece_id=YPXYWJWR2LMLCA4PFNLVHI4HBZ3ZP6VGIHCMAHVOADM5USZ3UJWA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148140441 (282h21m34.699610933s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9roAnE3zK4HaPFarGVAb8L3EfByGr22ufkLcGUKBeF9SqBA9): protocol: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53374->89.149.201.147:4010: use of closed network connection Nov 21 02:25:36.148153566 (282h21m34.699624027s) [noise=true , address=89.149.201.147:4010 , piece_id=X2Y6OVKDJOUA7DUWJAJAEN2FWYOFC3XXJPLNHCFQCXIV543RCIAA, seg_pos={0 0}, upload=248651]: Long tail cancellation due to optimal threshold Nov 21 02:25:36.148213310 (282h21m34.699683761s) [seg_pos={0 0}, upload=248651]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 02:25:36.148357414 (282h21m34.699827875s) [upload=248651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 02:25:36.148395076 (282h21m34.699865547s) [upload=248651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 02:25:36.406626090 (282h21m34.958096551s) [upload=248651]: Done uploading segments... Nov 21 02:25:36.406651539 (282h21m34.958122s) [upload=248651]: Upload finished. err: <nil> Duration: 6.26786166s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....................X.............X......X.............X...] duration: 4.537194353s trace: https://jaeger.dev.storj.io/trace/ce6552406b4751a FAIL (2025-11-21 03:01:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.537226725s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)