Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 11.760454634s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:17:24 UTC) orbiter warning: context deadline exceeded Duration: 11.760490031s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.0357113s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:17:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.035741196s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X..............X.....................X...] duration: 3.509143487s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:16:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.509174355s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 7.484726804s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:16:31 UTC) orbiter warning: context deadline exceeded Duration: 7.484756961s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 12.058545965s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:16:20 UTC) orbiter warning: context deadline exceeded Duration: 12.058559911s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.880992135s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:16:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.881020519s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................X..............X.....................X..] duration: 3.47105471s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:16:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.471089788s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.905114778s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:15:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.905148382s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.917049369s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:15:43 UTC) orbiter warning: context deadline exceeded Duration: 6.917081461s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................X..............X.....................X.] duration: 3.478678006s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:15:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.478725235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.387201825s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:15:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.387236241s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 11.581110415s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:14:59 UTC) orbiter warning: context deadline exceeded Duration: 11.581145753s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.810793932s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:14:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.810828528s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.969835164s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:14:05 UTC) orbiter warning: context deadline exceeded Duration: 6.969866433s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................X..............X.....................X] duration: 5.001392737s trace: https://jaeger.dev.storj.io/trace/5785071d81cfe2ae FAIL (2025-11-29 14:14:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001414909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.111881416s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:13:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.111992668s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.961396194s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:13:46 UTC) orbiter warning: context deadline exceeded Duration: 6.961426651s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................X..............X.....................] duration: 3.696093231s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:13:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.696110154s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.467167247s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:13:13 UTC) orbiter warning: context deadline exceeded Duration: 6.467197886s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.004262878s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:12:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.004293757s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 15.636270243s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:12:31 UTC) orbiter warning: context deadline exceeded Duration: 15.636312543s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.988046527s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:12:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.988078509s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................X..............X....................] duration: 3.55258901s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:12:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.552629234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.788879934s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:11:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.788915792s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 9.259773658s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:11:41 UTC) orbiter warning: context deadline exceeded Duration: 9.259804005s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X..............X...................] duration: 3.594681674s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:11:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.594706029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.147822747s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:10:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.14785078s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.034183149s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:10:45 UTC) orbiter warning: context deadline exceeded Duration: 6.034206083s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001754264s trace: https://jaeger.dev.storj.io/trace/61a1d0645153f16 FAIL (2025-11-29 14:10:38 UTC) orbiter warning: context deadline exceeded Duration: 30.001776405s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.141457002s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:10:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.141487369s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X..............X..................] duration: 3.871185069s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:09:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.871321327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.849982333s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:09:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.850008592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.86125528s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:09:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.861278334s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X..............X.................] duration: 3.684745069s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:08:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.684781018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.038069583s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:08:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.038229267s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X..............X................] duration: 3.686376576s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:08:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.686407957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.743814749s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:07:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.744439368s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.944390748s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:07:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.944423451s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X..............X...............] duration: 3.426194717s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:06:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.426227591s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.067132625s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:06:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.067162844s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X..............X..............] duration: 3.524895235s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:06:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524922918s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.712445116s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:05:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.712476044s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X..............X.............] duration: 3.424598976s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:05:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.424625627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X..............X............] duration: 5.001575722s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:04:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001850405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.163846634s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:04:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 6.163875298s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X..............X...........] duration: 3.576286721s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:03:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.576338079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.200641377s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:03:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 7.200671204s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X..............X..........] duration: 3.698446183s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:03:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.698477954s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................X..............X.........] duration: 5.002139161s trace: https://jaeger.dev.storj.io/trace/4e25c5dd7f4c018b FAIL (2025-11-29 14:02:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00216503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.824783998s trace: https://jaeger.dev.storj.io/trace/1c9128a6693258f6 FAIL (2025-11-29 14:02:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 9 13:48:17.211327572 (485h44m15.762798033s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.211902246 (485h44m15.763372717s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.212487982 (485h44m15.763958453s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214186295 (485h44m15.765656746s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.214478622 (485h44m15.765949083s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.215441214 (485h44m15.766911676s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216103165 (485h44m15.767573626s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.216785996 (485h44m15.768256447s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.217924283 (485h44m15.769394744s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.218158720 (485h44m15.769629181s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.219387229 (485h44m15.77085769s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.221891979 (485h44m15.77336244s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:17.223844847 (485h44m15.775315308s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Uploading piece... Nov 29 13:48:19.065199523 (485h44m17.616669985s) [upload=427723]: Next returned nil segment Nov 29 13:48:19.065231243 (485h44m17.616701725s) [upload=427723]: Waiting for error group managing segments... Nov 29 13:48:19.357892500 (485h44m17.909362961s) [noise=true , address=77.90.31.18:4806 , piece_id=AKNLWDX3CPMY3M3HK5HNYOX5BVNFBXKQHKFQNGFSWHWK7U4M6QQQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:19.388456861 (485h44m17.939927322s) [noise=true , address=45.91.116.7:4010 , piece_id=QROLFA4H6URFZOV7JIBBTQRAZ63UULB5SG4HI22MQUU4G4K5XJTQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:25.281875436 (485h44m23.833345897s) [noise=true , address=145.79.227.39:4014 , piece_id=ADAGAZOOHIYEWVR5SD72DUXJ7VKDFBUENBLYOGXGP3BABJLMP7FA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:40.222943408 (485h44m38.774413859s) [noise=true , address=145.79.227.25:4617 , piece_id=YWLSZU66P7UJCTNIPB2TX4ASE5YZDZ2RAKOWNCXH7EZMLDNTQ6ZQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: <nil> Nov 29 13:48:46.242376074 (485h44m44.793846525s) [noise=true , address=145.79.227.26:4653 , piece_id=LMUUYHV74LW6IBNN53NUHYWLIONHNZOS4HXRB57LW446UIAYZNKQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: rpc: dial tcp 145.79.227.26:4653: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242400701 (485h44m44.793871152s) [noise=true , address=145.79.227.22:4510 , piece_id=P2GNKZHFABTIXSO6ETV74SGXQ4TLNT5YWRG2JPR774G6YHP334QA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242542691 (485h44m44.794013162s) [noise=true , address=145.79.227.16:4911 , piece_id=X7DMLKH2LBR4PZ4O5WV2E5LFW6DCSEWD6WSVGMG6UQCUQKIIPINA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.242643744 (485h44m44.794114205s) [noise=true , address=145.79.227.40:4051 , piece_id=7PPPIQW7NHUH5HIAX4A53B33RIGC46VGEUJBIUCTCWNHASJFGQMQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: rpc: dial tcp 145.79.227.40:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.242742362 (485h44m44.794212813s) [noise=true , address=145.79.227.28:4727 , piece_id=MGI252VY4NPZIQUK3RYR6ANJYSZJKWOT57PJOOKX7TCKUR3ICYWA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32836->145.79.227.28:4727: use of closed network connection Nov 29 13:48:46.242822504 (485h44m44.794292965s) [noise=true , address=145.79.227.19:4079 , piece_id=CKETV5YXMWXZJOHDJLXLD6ZCCZXIN6IW5WU3KPMVIZJNAAYEEVVA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49268->145.79.227.19:4079: use of closed network connection Nov 29 13:48:46.242979943 (485h44m44.794450394s) [noise=true , address=145.79.227.42:4123 , piece_id=CEFUF55V25YAOX7TZFRIS6V3N7VVIKKNB4FTJVEHKASFJ3FS3YLQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36980->145.79.227.42:4123: use of closed network connection Nov 29 13:48:46.242998319 (485h44m44.79446877s) [noise=true , address=145.79.227.31:4799 , piece_id=SU6XXH63IS7R2TDJOF5T4K7FROKLXM5ZKNFAOJJLQKFEY7OHOINQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: rpc: dial tcp 145.79.227.31:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243136191 (485h44m44.794606642s) [noise=true , address=145.79.227.44:4194 , piece_id=2K2HSV336DG3JAI4MPGGPUUWZKNRSOYCBDSC4VZ4W2QVH23CDRSA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: expected piece hash; context deadline exceeded; EOF Nov 29 13:48:46.243153784 (485h44m44.794624245s) [noise=true , address=145.79.227.6:4257 , piece_id=S3HHHLMKRHBWDL4VF3EFZP5KCNN23WR7EPFEMAZHUY73MREMI7BA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: read tcp 188.166.225.170:55562->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243483853 (485h44m44.794954304s) [noise=true , address=145.79.227.30:4761 , piece_id=YEGWSQW4IYZFZZHHRIBXWUDGYVPMHE5QQY2NFE4A76MG2RLBFPNA, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: read tcp 188.166.225.170:34970->145.79.227.30:4761: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243612939 (485h44m44.795083401s) [noise=true , address=145.79.227.18:4006 , piece_id=T45HKNFKVYP2NSTANU3QVE2GTU5XGZJ4XUANET6EPYW6AFSJOAEQ, seg_pos={0 0}, upload=427723]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: read tcp 188.166.225.170:42644->145.79.227.18:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 29 13:48:46.243644820 (485h44m44.795115281s) [seg_pos={0 0}, upload=427723]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Nov 29 13:48:46.488678893 (485h44m45.040149344s) [upload=427723]: Done uploading segments... Nov 29 13:48:46.488730612 (485h44m45.040201063s) [upload=427723]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 5.824813904s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)