Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001754264s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:10:38 UTC) orbiter warning: context deadline exceeded Duration: 30.001776405s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.141457002s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:10:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.141487369s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X..............X..................] duration: 3.871185069s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:09:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.871321327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.849982333s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:09:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.850008592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.86125528s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:09:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.861278334s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X..............X.................] duration: 3.684745069s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:08:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.684781018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.038069583s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:08:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.038229267s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X..............X................] duration: 3.686376576s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:08:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.686407957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.743814749s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:07:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.744439368s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.944390748s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:07:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.944423451s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X..............X...............] duration: 3.426194717s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:06:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.426227591s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.067132625s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:06:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.067162844s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X..............X..............] duration: 3.524895235s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:06:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524922918s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.712445116s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:05:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.712476044s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X..............X.............] duration: 3.424598976s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:05:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.424625627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X..............X............] duration: 5.001575722s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:04:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001850405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.163846634s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:04:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.163875298s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X..............X...........] duration: 3.576286721s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:03:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.576338079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.200641377s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:03:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 7.200671204s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X..............X..........] duration: 3.698446183s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:03:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.698477954s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................X..............X.........] duration: 5.002139161s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:02:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00216503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.824783998s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:02:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.824813904s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................X..............X........] duration: 3.471407059s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:01:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.471435273s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................X..............X.......] duration: 3.390359579s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:01:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.390388805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.862453706s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:01:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.862585176s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................X..............X......] duration: 3.508796433s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:00:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.508829446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.753337735s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:00:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.753370608s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................X..............X.....] duration: 5.000552211s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 13:59:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000581004s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.719597107s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:59:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.719625822s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.966997688s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:58:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.967030551s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................X..............X....] duration: 3.454422787s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 13:58:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454446553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.893512039s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:58:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.893544011s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.821175395s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:57:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.821200382s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.69941969s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:57:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.699445839s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................X..............X...] duration: 3.320150794s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 13:57:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.320185911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.885556424s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:56:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.885608604s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................X..............X..] duration: 3.416607641s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 13:56:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.416652166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.79312481s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:56:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.793156099s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.736858347s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:55:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.736877985s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................X..............X.] duration: 3.617909679s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 13:55:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.618113255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.700233976s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:54:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.700261169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.989251891s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:54:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.989279775s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................................X..............X] duration: 5.001534577s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 13:54:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001560565s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.695521593s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:53:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.695553034s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 5.000688092s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:53:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000710123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.813633924s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:53:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.813662096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.752727367s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:52:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.752752476s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.699239255s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 13:52:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.699259675s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.461108006s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:52:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.461128725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.747151673s trace: https://jaeger.dev.storj.io/trace/312713bfa35274cf FAIL (2025-11-29 13:52:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.747186169s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)