Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 4.953715732s trace: https://jaeger.dev.storj.io/trace/51c602d400b15cea FAIL (2025-12-08 22:40:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.95374636s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.867919202s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 22:40:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.867947246s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.644053641s trace: https://jaeger.dev.storj.io/trace/72871f7c52601eaf FAIL (2025-12-08 22:40:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917385480 (78h34m23.788468516s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917994611 (78h34m23.789077666s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918445960 (78h34m23.789528995s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918888933 (78h34m23.789971978s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919333861 (78h34m23.790416906s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919736458 (78h34m23.790819503s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920148312 (78h34m23.791231347s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920546540 (78h34m23.791629585s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.921942007 (78h34m23.793025052s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.923813691 (78h34m23.794896736s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.925353112 (78h34m23.796436157s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.927009226 (78h34m23.798092261s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.928697040 (78h34m23.799780085s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.931092261 (78h34m23.802175306s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.932748024 (78h34m23.803831069s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.933733821 (78h34m23.804816856s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.935468164 (78h34m23.806551209s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:25.369562765 (78h34m25.24064581s) [upload=68635]: Next returned nil segment Dec 8 22:03:25.369578857 (78h34m25.240661892s) [upload=68635]: Waiting for error group managing segments... Dec 8 22:03:25.775620953 (78h34m25.646703999s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.798803277 (78h34m25.669886322s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.801642202 (78h34m25.672725237s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.802057232 (78h34m25.673140267s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816132514 (78h34m25.687215559s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816446331 (78h34m25.687529376s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.869308801 (78h34m25.740391846s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.873476507 (78h34m25.744559542s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.907465318 (78h34m25.778548354s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.926737626 (78h34m25.797820671s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.928011191 (78h34m25.799094226s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.930540317 (78h34m25.801623362s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.954520891 (78h34m25.825603926s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001264168 (78h34m25.872347203s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001283154 (78h34m25.872366189s) [seg_pos={0 0}, upload=68635]: Segment reached optimal threshold of 14 pieces. Dec 8 22:03:26.001442167 (78h34m25.872525202s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001448459 (78h34m25.872531494s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001703023 (78h34m25.872786068s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001710758 (78h34m25.872793803s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001867176 (78h34m25.872950231s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001872526 (78h34m25.872955581s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002060293 (78h34m25.873143328s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.002066315 (78h34m25.873149351s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002085732 (78h34m25.873168767s) [seg_pos={0 0}, upload=68635]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:03:26.002156887 (78h34m25.873239922s) [upload=68635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:03:26.002161526 (78h34m25.873244561s) [upload=68635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 22:04:50.380087807 (78h35m50.251170842s) [upload=68635]: Done uploading segments... Dec 8 22:04:50.380137331 (78h35m50.251220376s) [upload=68635]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.644089591s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.104532989s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:40:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.104572544s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [............X.X............X.....X..................X.......] duration: 3.579784971s trace: https://jaeger.dev.storj.io/trace/66132c8bfd2b5f8a FAIL (2025-12-08 22:40:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.579812532s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 4.994117394s trace: https://jaeger.dev.storj.io/trace/51c602d400b15cea FAIL (2025-12-08 22:40:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.994153124s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 527.678486ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5fc1020d8e24af13 FAIL (2025-12-08 22:40:03 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 527.752286ms core satellite eu1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [..........................X................X................] duration: 691.52225ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 22:39:58 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 691.550516ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X..X.........X...............................X.......] duration: 6.021062545s trace: https://jaeger.dev.storj.io/trace/71e1252488f598ce FAIL (2025-12-08 22:39:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.021167084s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.680091724s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:39:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.680123946s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [...........................X................X...............] duration: 687.371441ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 22:39:35 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 687.398673ms core satellite us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 466.570233ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5fc1020d8e24af13 FAIL (2025-12-08 22:39:24 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 466.587796ms core satellite eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.312657062s remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/5fc1020d8e24af13 FAIL (2025-12-08 22:39:23 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 1m0.312669005s core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.842032871s trace: https://jaeger.dev.storj.io/trace/72871f7c52601eaf FAIL (2025-12-08 22:39:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917385480 (78h34m23.788468516s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917994611 (78h34m23.789077666s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918445960 (78h34m23.789528995s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918888933 (78h34m23.789971978s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919333861 (78h34m23.790416906s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919736458 (78h34m23.790819503s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920148312 (78h34m23.791231347s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920546540 (78h34m23.791629585s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.921942007 (78h34m23.793025052s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.923813691 (78h34m23.794896736s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.925353112 (78h34m23.796436157s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.927009226 (78h34m23.798092261s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.928697040 (78h34m23.799780085s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.931092261 (78h34m23.802175306s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.932748024 (78h34m23.803831069s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.933733821 (78h34m23.804816856s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.935468164 (78h34m23.806551209s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:25.369562765 (78h34m25.24064581s) [upload=68635]: Next returned nil segment Dec 8 22:03:25.369578857 (78h34m25.240661892s) [upload=68635]: Waiting for error group managing segments... Dec 8 22:03:25.775620953 (78h34m25.646703999s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.798803277 (78h34m25.669886322s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.801642202 (78h34m25.672725237s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.802057232 (78h34m25.673140267s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816132514 (78h34m25.687215559s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816446331 (78h34m25.687529376s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.869308801 (78h34m25.740391846s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.873476507 (78h34m25.744559542s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.907465318 (78h34m25.778548354s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.926737626 (78h34m25.797820671s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.928011191 (78h34m25.799094226s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.930540317 (78h34m25.801623362s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.954520891 (78h34m25.825603926s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001264168 (78h34m25.872347203s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001283154 (78h34m25.872366189s) [seg_pos={0 0}, upload=68635]: Segment reached optimal threshold of 14 pieces. Dec 8 22:03:26.001442167 (78h34m25.872525202s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001448459 (78h34m25.872531494s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001703023 (78h34m25.872786068s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001710758 (78h34m25.872793803s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001867176 (78h34m25.872950231s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001872526 (78h34m25.872955581s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002060293 (78h34m25.873143328s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.002066315 (78h34m25.873149351s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002085732 (78h34m25.873168767s) [seg_pos={0 0}, upload=68635]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:03:26.002156887 (78h34m25.873239922s) [upload=68635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:03:26.002161526 (78h34m25.873244561s) [upload=68635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 22:04:50.380087807 (78h35m50.251170842s) [upload=68635]: Done uploading segments... Dec 8 22:04:50.380137331 (78h35m50.251220376s) [upload=68635]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.842064493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.850458515s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 22:39:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.850576198s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.307343749s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:39:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.307367534s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.087066671s trace: https://jaeger.dev.storj.io/trace/51c602d400b15cea FAIL (2025-12-08 22:39:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.087098031s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.............X.X............X.....X..................X......] duration: 3.462038269s trace: https://jaeger.dev.storj.io/trace/66132c8bfd2b5f8a FAIL (2025-12-08 22:39:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462068927s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.733041861s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:38:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.733072771s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X..X.........X...............................X......] duration: 5.97165201s trace: https://jaeger.dev.storj.io/trace/71e1252488f598ce FAIL (2025-12-08 22:38:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.971680293s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [............................X................X..............] duration: 1.258049566s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 22:38:37 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.258073822s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.781728049s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 22:38:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.781753899s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.843852526s trace: https://jaeger.dev.storj.io/trace/72871f7c52601eaf FAIL (2025-12-08 22:38:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917385480 (78h34m23.788468516s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917994611 (78h34m23.789077666s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918445960 (78h34m23.789528995s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918888933 (78h34m23.789971978s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919333861 (78h34m23.790416906s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919736458 (78h34m23.790819503s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920148312 (78h34m23.791231347s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920546540 (78h34m23.791629585s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.921942007 (78h34m23.793025052s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.923813691 (78h34m23.794896736s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.925353112 (78h34m23.796436157s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.927009226 (78h34m23.798092261s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.928697040 (78h34m23.799780085s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.931092261 (78h34m23.802175306s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.932748024 (78h34m23.803831069s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.933733821 (78h34m23.804816856s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.935468164 (78h34m23.806551209s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:25.369562765 (78h34m25.24064581s) [upload=68635]: Next returned nil segment Dec 8 22:03:25.369578857 (78h34m25.240661892s) [upload=68635]: Waiting for error group managing segments... Dec 8 22:03:25.775620953 (78h34m25.646703999s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.798803277 (78h34m25.669886322s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.801642202 (78h34m25.672725237s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.802057232 (78h34m25.673140267s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816132514 (78h34m25.687215559s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816446331 (78h34m25.687529376s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.869308801 (78h34m25.740391846s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.873476507 (78h34m25.744559542s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.907465318 (78h34m25.778548354s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.926737626 (78h34m25.797820671s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.928011191 (78h34m25.799094226s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.930540317 (78h34m25.801623362s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.954520891 (78h34m25.825603926s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001264168 (78h34m25.872347203s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001283154 (78h34m25.872366189s) [seg_pos={0 0}, upload=68635]: Segment reached optimal threshold of 14 pieces. Dec 8 22:03:26.001442167 (78h34m25.872525202s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001448459 (78h34m25.872531494s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001703023 (78h34m25.872786068s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001710758 (78h34m25.872793803s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001867176 (78h34m25.872950231s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001872526 (78h34m25.872955581s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002060293 (78h34m25.873143328s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.002066315 (78h34m25.873149351s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002085732 (78h34m25.873168767s) [seg_pos={0 0}, upload=68635]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:03:26.002156887 (78h34m25.873239922s) [upload=68635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:03:26.002161526 (78h34m25.873244561s) [upload=68635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 22:04:50.380087807 (78h35m50.251170842s) [upload=68635]: Done uploading segments... Dec 8 22:04:50.380137331 (78h35m50.251220376s) [upload=68635]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.84390759s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X..X.........X...............................X.....] duration: 5.792067909s trace: https://jaeger.dev.storj.io/trace/71e1252488f598ce FAIL (2025-12-08 22:38:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.792095029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 4.865464938s trace: https://jaeger.dev.storj.io/trace/51c602d400b15cea FAIL (2025-12-08 22:38:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.865560089s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.308947145s remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4ae585f031a538ca FAIL (2025-12-08 22:38:23 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 1m0.308973185s core satellite eu1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [.............................X................X.............] duration: 696.276277ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 22:38:17 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 696.301997ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.90725186s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 22:38:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.907284133s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..............X.X............X.....X..................X.....] duration: 3.497571081s trace: https://jaeger.dev.storj.io/trace/66132c8bfd2b5f8a FAIL (2025-12-08 22:38:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497594323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.828350046s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:38:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.828378561s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [..............................X................X............] duration: 695.391552ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 22:38:02 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 695.439984ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.009719848s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 22:37:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.009757138s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 4.914333694s trace: https://jaeger.dev.storj.io/trace/51c602d400b15cea FAIL (2025-12-08 22:37:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.914361396s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.882302841s trace: https://jaeger.dev.storj.io/trace/72871f7c52601eaf FAIL (2025-12-08 22:37:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917385480 (78h34m23.788468516s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917994611 (78h34m23.789077666s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918445960 (78h34m23.789528995s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918888933 (78h34m23.789971978s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919333861 (78h34m23.790416906s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919736458 (78h34m23.790819503s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920148312 (78h34m23.791231347s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920546540 (78h34m23.791629585s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.921942007 (78h34m23.793025052s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.923813691 (78h34m23.794896736s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.925353112 (78h34m23.796436157s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.927009226 (78h34m23.798092261s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.928697040 (78h34m23.799780085s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.931092261 (78h34m23.802175306s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.932748024 (78h34m23.803831069s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.933733821 (78h34m23.804816856s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.935468164 (78h34m23.806551209s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:25.369562765 (78h34m25.24064581s) [upload=68635]: Next returned nil segment Dec 8 22:03:25.369578857 (78h34m25.240661892s) [upload=68635]: Waiting for error group managing segments... Dec 8 22:03:25.775620953 (78h34m25.646703999s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.798803277 (78h34m25.669886322s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.801642202 (78h34m25.672725237s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.802057232 (78h34m25.673140267s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816132514 (78h34m25.687215559s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816446331 (78h34m25.687529376s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.869308801 (78h34m25.740391846s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.873476507 (78h34m25.744559542s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.907465318 (78h34m25.778548354s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.926737626 (78h34m25.797820671s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.928011191 (78h34m25.799094226s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.930540317 (78h34m25.801623362s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.954520891 (78h34m25.825603926s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001264168 (78h34m25.872347203s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001283154 (78h34m25.872366189s) [seg_pos={0 0}, upload=68635]: Segment reached optimal threshold of 14 pieces. Dec 8 22:03:26.001442167 (78h34m25.872525202s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001448459 (78h34m25.872531494s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001703023 (78h34m25.872786068s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001710758 (78h34m25.872793803s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001867176 (78h34m25.872950231s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001872526 (78h34m25.872955581s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002060293 (78h34m25.873143328s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.002066315 (78h34m25.873149351s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002085732 (78h34m25.873168767s) [seg_pos={0 0}, upload=68635]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:03:26.002156887 (78h34m25.873239922s) [upload=68635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:03:26.002161526 (78h34m25.873244561s) [upload=68635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 22:04:50.380087807 (78h35m50.251170842s) [upload=68635]: Done uploading segments... Dec 8 22:04:50.380137331 (78h35m50.251220376s) [upload=68635]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.88233306s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...............X.X............X.....X..................X....] duration: 3.431633193s trace: https://jaeger.dev.storj.io/trace/66132c8bfd2b5f8a FAIL (2025-12-08 22:37:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.431655095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.867965823s trace: https://jaeger.dev.storj.io/trace/72871f7c52601eaf FAIL (2025-12-08 22:37:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917385480 (78h34m23.788468516s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917994611 (78h34m23.789077666s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918445960 (78h34m23.789528995s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918888933 (78h34m23.789971978s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919333861 (78h34m23.790416906s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919736458 (78h34m23.790819503s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920148312 (78h34m23.791231347s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920546540 (78h34m23.791629585s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.921942007 (78h34m23.793025052s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.923813691 (78h34m23.794896736s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.925353112 (78h34m23.796436157s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.927009226 (78h34m23.798092261s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.928697040 (78h34m23.799780085s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.931092261 (78h34m23.802175306s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.932748024 (78h34m23.803831069s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.933733821 (78h34m23.804816856s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.935468164 (78h34m23.806551209s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:25.369562765 (78h34m25.24064581s) [upload=68635]: Next returned nil segment Dec 8 22:03:25.369578857 (78h34m25.240661892s) [upload=68635]: Waiting for error group managing segments... Dec 8 22:03:25.775620953 (78h34m25.646703999s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.798803277 (78h34m25.669886322s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.801642202 (78h34m25.672725237s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.802057232 (78h34m25.673140267s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816132514 (78h34m25.687215559s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816446331 (78h34m25.687529376s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.869308801 (78h34m25.740391846s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.873476507 (78h34m25.744559542s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.907465318 (78h34m25.778548354s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.926737626 (78h34m25.797820671s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.928011191 (78h34m25.799094226s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.930540317 (78h34m25.801623362s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.954520891 (78h34m25.825603926s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001264168 (78h34m25.872347203s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001283154 (78h34m25.872366189s) [seg_pos={0 0}, upload=68635]: Segment reached optimal threshold of 14 pieces. Dec 8 22:03:26.001442167 (78h34m25.872525202s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001448459 (78h34m25.872531494s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001703023 (78h34m25.872786068s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001710758 (78h34m25.872793803s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001867176 (78h34m25.872950231s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001872526 (78h34m25.872955581s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002060293 (78h34m25.873143328s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.002066315 (78h34m25.873149351s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002085732 (78h34m25.873168767s) [seg_pos={0 0}, upload=68635]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:03:26.002156887 (78h34m25.873239922s) [upload=68635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:03:26.002161526 (78h34m25.873244561s) [upload=68635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 22:04:50.380087807 (78h35m50.251170842s) [upload=68635]: Done uploading segments... Dec 8 22:04:50.380137331 (78h35m50.251220376s) [upload=68635]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.867993546s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X..X.........X...............................X....] duration: 5.699451603s trace: https://jaeger.dev.storj.io/trace/71e1252488f598ce FAIL (2025-12-08 22:37:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.699477933s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.664997664s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:37:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.665026789s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [...............................X................X...........] duration: 686.711374ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 22:37:02 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 686.761368ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.770108205s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 22:36:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.770131738s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [................X.X............X.....X..................X...] duration: 3.547346054s trace: https://jaeger.dev.storj.io/trace/66132c8bfd2b5f8a FAIL (2025-12-08 22:36:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.547374769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 4.688547801s trace: https://jaeger.dev.storj.io/trace/51c602d400b15cea FAIL (2025-12-08 22:36:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.688574531s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........X..X.........X...............................X...] duration: 5.861811936s trace: https://jaeger.dev.storj.io/trace/71e1252488f598ce FAIL (2025-12-08 22:36:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.861832886s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [................................X................X..........] duration: 690.281672ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 22:36:34 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 690.304977ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.................X.X............X.....X..................X..] duration: 3.417166544s trace: https://jaeger.dev.storj.io/trace/66132c8bfd2b5f8a FAIL (2025-12-08 22:36:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.417193746s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.831012055s trace: https://jaeger.dev.storj.io/trace/2194d46470449815 FAIL (2025-12-08 22:36:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.831043295s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.731820367s trace: https://jaeger.dev.storj.io/trace/72871f7c52601eaf FAIL (2025-12-08 22:36:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917385480 (78h34m23.788468516s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917994611 (78h34m23.789077666s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918445960 (78h34m23.789528995s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918888933 (78h34m23.789971978s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919333861 (78h34m23.790416906s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919736458 (78h34m23.790819503s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920148312 (78h34m23.791231347s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920546540 (78h34m23.791629585s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.921942007 (78h34m23.793025052s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.923813691 (78h34m23.794896736s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.925353112 (78h34m23.796436157s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.927009226 (78h34m23.798092261s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.928697040 (78h34m23.799780085s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.931092261 (78h34m23.802175306s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.932748024 (78h34m23.803831069s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.933733821 (78h34m23.804816856s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.935468164 (78h34m23.806551209s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:25.369562765 (78h34m25.24064581s) [upload=68635]: Next returned nil segment Dec 8 22:03:25.369578857 (78h34m25.240661892s) [upload=68635]: Waiting for error group managing segments... Dec 8 22:03:25.775620953 (78h34m25.646703999s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.798803277 (78h34m25.669886322s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.801642202 (78h34m25.672725237s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.802057232 (78h34m25.673140267s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816132514 (78h34m25.687215559s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816446331 (78h34m25.687529376s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.869308801 (78h34m25.740391846s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.873476507 (78h34m25.744559542s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.907465318 (78h34m25.778548354s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.926737626 (78h34m25.797820671s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.928011191 (78h34m25.799094226s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.930540317 (78h34m25.801623362s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.954520891 (78h34m25.825603926s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001264168 (78h34m25.872347203s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001283154 (78h34m25.872366189s) [seg_pos={0 0}, upload=68635]: Segment reached optimal threshold of 14 pieces. Dec 8 22:03:26.001442167 (78h34m25.872525202s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001448459 (78h34m25.872531494s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001703023 (78h34m25.872786068s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001710758 (78h34m25.872793803s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001867176 (78h34m25.872950231s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001872526 (78h34m25.872955581s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002060293 (78h34m25.873143328s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.002066315 (78h34m25.873149351s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002085732 (78h34m25.873168767s) [seg_pos={0 0}, upload=68635]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:03:26.002156887 (78h34m25.873239922s) [upload=68635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:03:26.002161526 (78h34m25.873244561s) [upload=68635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 22:04:50.380087807 (78h35m50.251170842s) [upload=68635]: Done uploading segments... Dec 8 22:04:50.380137331 (78h35m50.251220376s) [upload=68635]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.731847638s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.750598742s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:36:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.750658166s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............X..X.........X...............................X..] duration: 5.934569819s trace: https://jaeger.dev.storj.io/trace/71e1252488f598ce FAIL (2025-12-08 22:36:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.93474845s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [.................................X................X.........] duration: 692.449208ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4045469d14ef4ed6 FAIL (2025-12-08 22:36:11 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 692.475389ms core satellite us1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)