Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.893512039s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:58:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.893544011s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.821175395s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:57:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.821200382s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.69941969s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:57:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.699445839s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................X..............X...] duration: 3.320150794s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 13:57:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.320185911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.885556424s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:56:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.885608604s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................X..............X..] duration: 3.416607641s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 13:56:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.416652166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.79312481s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:56:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.793156099s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.736858347s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:55:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.736877985s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................X..............X.] duration: 3.617909679s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 13:55:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.618113255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.700233976s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:54:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.700261169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.989251891s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:54:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.989279775s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................................X..............X] duration: 5.001534577s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 13:54:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001560565s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.695521593s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:53:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.695553034s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 5.000688092s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:53:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000710123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.813633924s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:53:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.813662096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.752727367s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:52:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.752752476s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.699239255s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:52:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.699259675s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.461108006s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:52:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.461128725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.747151673s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:52:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.747186169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.842880707s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:51:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.842909301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.725444863s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:51:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.72550178s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.494204988s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:51:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.494344374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.823074591s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:50:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.823109487s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.480483439s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:50:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.480513398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.829835409s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:50:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.829851008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.733570568s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:49:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.73359738s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.428010141s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:49:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.428123637s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.776408249s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:49:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.776436723s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.477084868s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:49:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.477104667s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.248231094s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:48:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.248253177s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 4.05590552s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:48:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.055932821s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.458780784s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:47:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.458807785s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.50569813s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:46:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.505715795s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.613246375s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:46:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.613273517s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 4.8123649s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:45:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.812389557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.446330954s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:44:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.44636095s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.655454788s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:43:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.655482211s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.000861907s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:42:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000890382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.870985833s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 09:18:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.871015049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.752093371s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 09:17:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.752128499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.848466698s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 09:17:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.848498998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.772594371s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 09:16:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.772624579s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.830003138s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 09:15:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.830031903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.820219218s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 09:14:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.820247623s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.925665253s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 09:14:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.925794319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.909301378s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 09:13:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.909320585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.800886131s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 09:12:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.800921479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.843740862s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 09:11:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.843772323s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.745525881s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 09:10:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.745547102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.861538041s trace: https://jaeger.dev.storj.io/trace/52a65a7acf41d7b8 FAIL (2025-11-29 09:10:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 7968805 (480h29m51.879439256s) [upload=423109]: Uploading segments... Nov 29 08:33:53.328022889 (480h29m51.87949338s) [upload=423109]: Got next segment. Inline: false Nov 29 08:33:53.328063486 (480h29m51.879533947s) [upload=423109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 29 08:33:53.328069577 (480h29m51.879540028s) [upload=423109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 29 08:33:53.613447611 (480h29m52.164918072s) [seg_pos={0 0}, upload=423109]: Begin upload segment... Nov 29 08:33:53.613514859 (480h29m52.16498532s) [seg_pos={0 0}, upload=423109]: Done begin upload segment. Nov 29 08:33:53.613569824 (480h29m52.165040286s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.613785064 (480h29m52.165255535s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614443047 (480h29m52.165913508s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.614858719 (480h29m52.16632918s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615328233 (480h29m52.166798694s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615341037 (480h29m52.166811498s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615638364 (480h29m52.167108825s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.615951711 (480h29m52.167422172s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616173763 (480h29m52.167644214s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616357012 (480h29m52.167827463s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616675709 (480h29m52.16814617s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.616733318 (480h29m52.168203779s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.617166694 (480h29m52.168637155s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.619882526 (480h29m52.171352987s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620424307 (480h29m52.171894768s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:53.620863123 (480h29m52.172333584s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Uploading piece... Nov 29 08:33:55.385669567 (480h29m53.937140018s) [upload=423109]: Next returned nil segment Nov 29 08:33:55.385687320 (480h29m53.937157781s) [upload=423109]: Waiting for error group managing segments... Nov 29 08:33:55.666465206 (480h29m54.217935667s) [noise=true , address=77.90.31.10:4520 , piece_id=DJ5NMSCWZF7HIWLNOZB4PIDBPUEDBINPUGM4ZUPWVDM4JRXMOEPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.239433679 (480h29m54.79090414s) [noise=true , address=45.91.116.13:4154 , piece_id=UQDEOHFN2ZK5P5ETWWTZSASB46D7TCMPS7JJWJGMDWDXG4TA2OOA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:33:56.686026739 (480h29m55.2374972s) [noise=true , address=45.91.116.4:4938 , piece_id=QL3OTADUAYN7IZFLO66JZBZFX6ROWVEHRRDPKZ3SVMOJWIP7TQTA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:00.122483289 (480h29m58.67395375s) [noise=true , address=145.79.227.8:4330 , piece_id=74FRXJQ6HTWHFYV57T6EIUNFKGKQG7QQHJO2O3TKXGYSBAPDEF6A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.136707860 (480h29m59.688178311s) [noise=true , address=145.79.227.42:4121 , piece_id=F6K3AQYIZ2XZ25YO3AMDNV7O4VCVP7LGOGQ2NDTYWWHGVMQXFBDA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:01.973016304 (480h30m0.524486766s) [noise=true , address=145.79.227.30:4763 , piece_id=ZFRA4TR5ZCYSBNJHRHHNKD7L4DZWROHCUIEN7JILM6HJCDESDHUQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:02.693901419 (480h30m1.245371871s) [noise=true , address=145.79.227.13:4803 , piece_id=XSPXCBPXLJSHTTERHROI5ENGVUPX3ZVQCRR4ZOLIK63HQX3ENWPQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:05.881771091 (480h30m4.433241562s) [noise=true , address=145.79.227.23:4547 , piece_id=KQ5QNHQULZK2VP64UMVHDC5TEWCR323ZCQ2645X4BHVN4U6AGEEA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.562831568 (480h30m17.114302029s) [noise=true , address=145.79.227.34:4906 , piece_id=UTY7CZEMATUJKPJQTJRGNUDXI5SF3GK44B77UWRF5XMO5Y3AKS3Q, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:18.693176069 (480h30m17.24464653s) [noise=true , address=145.79.227.29:4941 , piece_id=Y7YMFQUIQGUM4NNEZFU5LNVIRYEIL35WZSTBWZW7QDTTEF372LDQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: <nil> Nov 29 08:34:22.519375537 (480h30m21.070845998s) [noise=true , address=145.79.227.39:4014 , piece_id=55HOWRRUIYTFZDCRYJ5MHVQMJTVOV4G2JT2QIF7KCJH6LCCIPB5A, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33230->145.79.227.39:4014: use of closed network connection Nov 29 08:34:22.519448476 (480h30m21.070918947s) [noise=true , address=145.79.227.15:4875 , piece_id=HIQFVXKLCUAWBNCN2WTML7HRGNE4QQ26JUIADRCKEDL45FDAZHQQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60082->145.79.227.15:4875: use of closed network connection Nov 29 08:34:22.519611146 (480h30m21.071081607s) [noise=true , address=145.79.227.31:4797 , piece_id=D4ZR63M5E5BWYX7B4BQ7ZXPJD5JEL4RZ23O2IOYFWBIJTPKVX2MA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519682652 (480h30m21.071153133s) [noise=true , address=145.79.227.32:4836 , piece_id=CQ45JBLGRTJKBL6QWEJJ2EVZH3XZRPEAKWL4LEO6CYT3CYGJT6LQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51052->145.79.227.32:4836: use of closed network connection Nov 29 08:34:22.519828991 (480h30m21.071299472s) [noise=true , address=145.79.227.2:4115 , piece_id=PDIVDMASZ2L23JZU7RVJNJOIMJJSBNRIE6WDKLN46GDG7IGWUODA, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 08:34:22.519952717 (480h30m21.071423188s) [noise=true , address=145.79.227.11:4730 , piece_id=JPD2E3PEKUC5U2YXIXRGDMUGAUD53WC2JTPOUNNLFBHSVY44AFNQ, seg_pos={0 0}, upload=423109]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.11:4730: use of closed network connection Nov 29 08:34:22.519987823 (480h30m21.071458275s) [seg_pos={0 0}, upload=423109]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 29 08:34:22.768772489 (480h30m21.32024295s) [upload=423109]: Done uploading segments... Nov 29 08:34:22.768822204 (480h30m21.320292665s) [upload=423109]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.861570622s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)