Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X..............X.....................X...................] duration: 3.430637295s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:28:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.4306662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.812976271s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:28:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.812997972s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.822611027s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:28:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.82264345s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.528578692s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:27:57 UTC) orbiter warning: context deadline exceeded Duration: 6.528613929s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X..............X.....................X..................] duration: 3.500830321s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:27:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.500858894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.74624848s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:27:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.746279129s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....X..............X.....................X.................] duration: 3.520561738s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:26:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520581916s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.296410473s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:26:46 UTC) orbiter warning: context deadline exceeded Duration: 6.29644037s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.741195205s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:26:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.741227014s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X..............X.....................X................] duration: 3.576068105s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:26:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.576106819s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.503061301s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:25:58 UTC) orbiter warning: context deadline exceeded Duration: 6.503089765s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......X..............X.....................X...............] duration: 3.496717397s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:25:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.496745781s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.002499164s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:25:37 UTC) orbiter warning: context deadline exceeded Duration: 6.002531756s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.781016843s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:25:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.781070044s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........X..............X.....................X..............] duration: 3.743784258s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:25:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.743825667s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 9.077524201s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:25:02 UTC) orbiter warning: context deadline exceeded Duration: 9.077554538s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........X..............X.....................X.............] duration: 3.509456062s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:24:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.509479146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.887667452s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:24:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.887698192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.907464492s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:24:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.907493958s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........X..............X.....................X............] duration: 3.539597226s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:24:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.539627342s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.024976526s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:23:45 UTC) orbiter warning: context deadline exceeded Duration: 6.025020669s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.797561218s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:23:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.797582568s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.160046958s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:23:18 UTC) orbiter warning: context deadline exceeded Duration: 6.160226399s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........X..............X.....................X...........] duration: 3.365999451s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:23:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.366040551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.77385703s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:22:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.773876937s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 10.1501086s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:22:29 UTC) orbiter warning: context deadline exceeded Duration: 10.150137684s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............X..............X.....................X..........] duration: 3.651357157s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:22:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.651385252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.769278007s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:22:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.769311249s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 12.657956881s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:22:08 UTC) orbiter warning: context deadline exceeded Duration: 12.658001496s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X..............X.....................X.........] duration: 3.514750235s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:21:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.5147789s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.9108655s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:21:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.910897521s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.921040093s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:21:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.921069069s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 9.460034095s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:21:00 UTC) orbiter warning: context deadline exceeded Duration: 9.460116022s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............X..............X.....................X........] duration: 3.464782196s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:20:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.464817764s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 8.368945684s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:20:09 UTC) orbiter warning: context deadline exceeded Duration: 8.368964209s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.899768985s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:20:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.899801238s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............X..............X.....................X.......] duration: 4.868789943s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:19:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.86882071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.598148796s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:19:37 UTC) orbiter warning: context deadline exceeded Duration: 6.598165067s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.76795047s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:19:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.767985658s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................X..............X.....................X......] duration: 3.573710456s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:19:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.573739442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.774745199s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:18:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.774775607s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 10.864268104s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:18:30 UTC) orbiter warning: context deadline exceeded Duration: 10.864294123s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................X..............X.....................X.....] duration: 3.417407536s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:18:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.417434416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.402225522s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:18:02 UTC) orbiter warning: context deadline exceeded Duration: 6.402267472s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.040888488s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:17:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.040913395s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................X..............X.....................X....] duration: 3.352987809s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:17:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.353016625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 11.760454634s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:17:24 UTC) orbiter warning: context deadline exceeded Duration: 11.760490031s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.0357113s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:17:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.035741196s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X..............X.....................X...] duration: 3.509143487s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:16:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.509174355s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 7.484726804s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:16:31 UTC) orbiter warning: context deadline exceeded Duration: 7.484756961s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)