Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 865930 reports (Page 11675 of 17319)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 690.259849ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/f605f5110d8504 FAIL (2025-12-08 22:24:51 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 690.283173ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...............X..............................] duration: 5.715750463s trace: https://jaeger.dev.storj.io/trace/72871f7c52601eaf FAIL (2025-12-08 22:24:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917385480 (78h34m23.788468516s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917994611 (78h34m23.789077666s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918445960 (78h34m23.789528995s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918888933 (78h34m23.789971978s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919333861 (78h34m23.790416906s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919736458 (78h34m23.790819503s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920148312 (78h34m23.791231347s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920546540 (78h34m23.791629585s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.921942007 (78h34m23.793025052s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.923813691 (78h34m23.794896736s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.925353112 (78h34m23.796436157s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.927009226 (78h34m23.798092261s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.928697040 (78h34m23.799780085s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.931092261 (78h34m23.802175306s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.932748024 (78h34m23.803831069s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.933733821 (78h34m23.804816856s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.935468164 (78h34m23.806551209s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:25.369562765 (78h34m25.24064581s) [upload=68635]: Next returned nil segment Dec 8 22:03:25.369578857 (78h34m25.240661892s) [upload=68635]: Waiting for error group managing segments... Dec 8 22:03:25.775620953 (78h34m25.646703999s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.798803277 (78h34m25.669886322s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.801642202 (78h34m25.672725237s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.802057232 (78h34m25.673140267s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816132514 (78h34m25.687215559s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816446331 (78h34m25.687529376s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.869308801 (78h34m25.740391846s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.873476507 (78h34m25.744559542s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.907465318 (78h34m25.778548354s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.926737626 (78h34m25.797820671s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.928011191 (78h34m25.799094226s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.930540317 (78h34m25.801623362s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.954520891 (78h34m25.825603926s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001264168 (78h34m25.872347203s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001283154 (78h34m25.872366189s) [seg_pos={0 0}, upload=68635]: Segment reached optimal threshold of 14 pieces. Dec 8 22:03:26.001442167 (78h34m25.872525202s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001448459 (78h34m25.872531494s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001703023 (78h34m25.872786068s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001710758 (78h34m25.872793803s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001867176 (78h34m25.872950231s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001872526 (78h34m25.872955581s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002060293 (78h34m25.873143328s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.002066315 (78h34m25.873149351s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002085732 (78h34m25.873168767s) [seg_pos={0 0}, upload=68635]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:03:26.002156887 (78h34m25.873239922s) [upload=68635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:03:26.002161526 (78h34m25.873244561s) [upload=68635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 22:04:50.380087807 (78h35m50.251170842s) [upload=68635]: Done uploading segments... Dec 8 22:04:50.380137331 (78h35m50.251220376s) [upload=68635]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.715775131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.905914309s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:24:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.905939828s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 4.852753132s trace: https://jaeger.dev.storj.io/trace/51c602d400b15cea FAIL (2025-12-08 22:24:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.852772559s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.XXX............................X.X............X.....X......] duration: 3.497545535s trace: https://jaeger.dev.storj.io/trace/5cdb098fa00c6780 FAIL (2025-12-08 22:24:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497725095s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.793673042s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:24:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.793697059s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........................X..X.........X...................] duration: 5.72784315s trace: https://jaeger.dev.storj.io/trace/1e577f05655b28db FAIL (2025-12-08 22:24:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.727916519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...............X.............................] duration: 5.58617568s trace: https://jaeger.dev.storj.io/trace/72871f7c52601eaf FAIL (2025-12-08 22:24:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917385480 (78h34m23.788468516s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917994611 (78h34m23.789077666s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918445960 (78h34m23.789528995s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918888933 (78h34m23.789971978s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919333861 (78h34m23.790416906s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919736458 (78h34m23.790819503s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920148312 (78h34m23.791231347s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920546540 (78h34m23.791629585s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.921942007 (78h34m23.793025052s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.923813691 (78h34m23.794896736s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.925353112 (78h34m23.796436157s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.927009226 (78h34m23.798092261s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.928697040 (78h34m23.799780085s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.931092261 (78h34m23.802175306s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.932748024 (78h34m23.803831069s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.933733821 (78h34m23.804816856s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.935468164 (78h34m23.806551209s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:25.369562765 (78h34m25.24064581s) [upload=68635]: Next returned nil segment Dec 8 22:03:25.369578857 (78h34m25.240661892s) [upload=68635]: Waiting for error group managing segments... Dec 8 22:03:25.775620953 (78h34m25.646703999s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.798803277 (78h34m25.669886322s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.801642202 (78h34m25.672725237s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.802057232 (78h34m25.673140267s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816132514 (78h34m25.687215559s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816446331 (78h34m25.687529376s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.869308801 (78h34m25.740391846s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.873476507 (78h34m25.744559542s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.907465318 (78h34m25.778548354s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.926737626 (78h34m25.797820671s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.928011191 (78h34m25.799094226s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.930540317 (78h34m25.801623362s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.954520891 (78h34m25.825603926s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001264168 (78h34m25.872347203s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001283154 (78h34m25.872366189s) [seg_pos={0 0}, upload=68635]: Segment reached optimal threshold of 14 pieces. Dec 8 22:03:26.001442167 (78h34m25.872525202s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001448459 (78h34m25.872531494s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001703023 (78h34m25.872786068s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001710758 (78h34m25.872793803s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001867176 (78h34m25.872950231s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001872526 (78h34m25.872955581s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002060293 (78h34m25.873143328s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.002066315 (78h34m25.873149351s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002085732 (78h34m25.873168767s) [seg_pos={0 0}, upload=68635]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:03:26.002156887 (78h34m25.873239922s) [upload=68635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:03:26.002161526 (78h34m25.873244561s) [upload=68635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 22:04:50.380087807 (78h35m50.251170842s) [upload=68635]: Done uploading segments... Dec 8 22:04:50.380137331 (78h35m50.251220376s) [upload=68635]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.586204827s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 687.294412ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/f605f5110d8504 FAIL (2025-12-08 22:24:04 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 687.342433ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..XXX............................X.X............X.....X.....] duration: 3.439365589s trace: https://jaeger.dev.storj.io/trace/5cdb098fa00c6780 FAIL (2025-12-08 22:24:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.43939773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.031520529s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 22:23:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.031545987s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............................X..X.........X..................] duration: 5.806627731s trace: https://jaeger.dev.storj.io/trace/1e577f05655b28db FAIL (2025-12-08 22:23:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.806752348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 4.939438305s trace: https://jaeger.dev.storj.io/trace/51c602d400b15cea FAIL (2025-12-08 22:23:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.939463875s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.79017379s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:23:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.790219137s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.032800888s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 22:23:27 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.032830173s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...XXX............................X.X............X.....X....] duration: 3.451918222s trace: https://jaeger.dev.storj.io/trace/5cdb098fa00c6780 FAIL (2025-12-08 22:23:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.451947839s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.676294484s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:23:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.676327677s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............................X..X.........X.................] duration: 5.780299628s trace: https://jaeger.dev.storj.io/trace/1e577f05655b28db FAIL (2025-12-08 22:22:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.780326188s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 693.108471ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/f605f5110d8504 FAIL (2025-12-08 22:22:51 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 693.141726ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...............X............................] duration: 5.674059067s trace: https://jaeger.dev.storj.io/trace/72871f7c52601eaf FAIL (2025-12-08 22:22:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917385480 (78h34m23.788468516s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917994611 (78h34m23.789077666s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918445960 (78h34m23.789528995s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918888933 (78h34m23.789971978s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919333861 (78h34m23.790416906s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919736458 (78h34m23.790819503s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920148312 (78h34m23.791231347s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920546540 (78h34m23.791629585s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.921942007 (78h34m23.793025052s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.923813691 (78h34m23.794896736s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.925353112 (78h34m23.796436157s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.927009226 (78h34m23.798092261s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.928697040 (78h34m23.799780085s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.931092261 (78h34m23.802175306s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.932748024 (78h34m23.803831069s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.933733821 (78h34m23.804816856s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.935468164 (78h34m23.806551209s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:25.369562765 (78h34m25.24064581s) [upload=68635]: Next returned nil segment Dec 8 22:03:25.369578857 (78h34m25.240661892s) [upload=68635]: Waiting for error group managing segments... Dec 8 22:03:25.775620953 (78h34m25.646703999s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.798803277 (78h34m25.669886322s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.801642202 (78h34m25.672725237s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.802057232 (78h34m25.673140267s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816132514 (78h34m25.687215559s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816446331 (78h34m25.687529376s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.869308801 (78h34m25.740391846s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.873476507 (78h34m25.744559542s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.907465318 (78h34m25.778548354s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.926737626 (78h34m25.797820671s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.928011191 (78h34m25.799094226s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.930540317 (78h34m25.801623362s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.954520891 (78h34m25.825603926s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001264168 (78h34m25.872347203s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001283154 (78h34m25.872366189s) [seg_pos={0 0}, upload=68635]: Segment reached optimal threshold of 14 pieces. Dec 8 22:03:26.001442167 (78h34m25.872525202s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001448459 (78h34m25.872531494s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001703023 (78h34m25.872786068s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001710758 (78h34m25.872793803s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001867176 (78h34m25.872950231s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001872526 (78h34m25.872955581s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002060293 (78h34m25.873143328s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.002066315 (78h34m25.873149351s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002085732 (78h34m25.873168767s) [seg_pos={0 0}, upload=68635]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:03:26.002156887 (78h34m25.873239922s) [upload=68635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:03:26.002161526 (78h34m25.873244561s) [upload=68635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 22:04:50.380087807 (78h35m50.251170842s) [upload=68635]: Done uploading segments... Dec 8 22:04:50.380137331 (78h35m50.251220376s) [upload=68635]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.674076869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 4.822637387s trace: https://jaeger.dev.storj.io/trace/51c602d400b15cea FAIL (2025-12-08 22:22:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.822662013s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X...............X...........................] duration: 5.743615514s trace: https://jaeger.dev.storj.io/trace/72871f7c52601eaf FAIL (2025-12-08 22:22:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917385480 (78h34m23.788468516s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917994611 (78h34m23.789077666s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918445960 (78h34m23.789528995s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918888933 (78h34m23.789971978s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919333861 (78h34m23.790416906s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919736458 (78h34m23.790819503s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920148312 (78h34m23.791231347s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920546540 (78h34m23.791629585s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.921942007 (78h34m23.793025052s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.923813691 (78h34m23.794896736s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.925353112 (78h34m23.796436157s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.927009226 (78h34m23.798092261s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.928697040 (78h34m23.799780085s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.931092261 (78h34m23.802175306s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.932748024 (78h34m23.803831069s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.933733821 (78h34m23.804816856s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.935468164 (78h34m23.806551209s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:25.369562765 (78h34m25.24064581s) [upload=68635]: Next returned nil segment Dec 8 22:03:25.369578857 (78h34m25.240661892s) [upload=68635]: Waiting for error group managing segments... Dec 8 22:03:25.775620953 (78h34m25.646703999s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.798803277 (78h34m25.669886322s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.801642202 (78h34m25.672725237s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.802057232 (78h34m25.673140267s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816132514 (78h34m25.687215559s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816446331 (78h34m25.687529376s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.869308801 (78h34m25.740391846s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.873476507 (78h34m25.744559542s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.907465318 (78h34m25.778548354s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.926737626 (78h34m25.797820671s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.928011191 (78h34m25.799094226s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.930540317 (78h34m25.801623362s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.954520891 (78h34m25.825603926s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001264168 (78h34m25.872347203s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001283154 (78h34m25.872366189s) [seg_pos={0 0}, upload=68635]: Segment reached optimal threshold of 14 pieces. Dec 8 22:03:26.001442167 (78h34m25.872525202s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001448459 (78h34m25.872531494s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001703023 (78h34m25.872786068s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001710758 (78h34m25.872793803s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001867176 (78h34m25.872950231s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001872526 (78h34m25.872955581s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002060293 (78h34m25.873143328s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.002066315 (78h34m25.873149351s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002085732 (78h34m25.873168767s) [seg_pos={0 0}, upload=68635]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:03:26.002156887 (78h34m25.873239922s) [upload=68635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:03:26.002161526 (78h34m25.873244561s) [upload=68635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 22:04:50.380087807 (78h35m50.251170842s) [upload=68635]: Done uploading segments... Dec 8 22:04:50.380137331 (78h35m50.251220376s) [upload=68635]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.743646542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.013859208s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 22:22:23 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.013883634s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............................X..X.........X................] duration: 5.738958596s trace: https://jaeger.dev.storj.io/trace/1e577f05655b28db FAIL (2025-12-08 22:22:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.738985196s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 689.93549ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/f605f5110d8504 FAIL (2025-12-08 22:22:17 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 689.959517ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.125407769s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 22:22:04 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.125437306s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.741012139s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:21:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.741038337s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....XXX............................X.X............X.....X...] duration: 3.46888543s trace: https://jaeger.dev.storj.io/trace/5cdb098fa00c6780 FAIL (2025-12-08 22:21:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.468910038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............................X..X.........X...............] duration: 6.007125494s trace: https://jaeger.dev.storj.io/trace/1e577f05655b28db FAIL (2025-12-08 22:21:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.007231186s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 694.59836ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/f605f5110d8504 FAIL (2025-12-08 22:21:47 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 694.625332ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.109788473s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:21:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.109816549s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X...............X..........................] duration: 5.600042957s trace: https://jaeger.dev.storj.io/trace/72871f7c52601eaf FAIL (2025-12-08 22:21:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917385480 (78h34m23.788468516s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917994611 (78h34m23.789077666s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918445960 (78h34m23.789528995s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918888933 (78h34m23.789971978s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919333861 (78h34m23.790416906s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919736458 (78h34m23.790819503s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920148312 (78h34m23.791231347s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920546540 (78h34m23.791629585s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.921942007 (78h34m23.793025052s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.923813691 (78h34m23.794896736s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.925353112 (78h34m23.796436157s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.927009226 (78h34m23.798092261s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.928697040 (78h34m23.799780085s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.931092261 (78h34m23.802175306s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.932748024 (78h34m23.803831069s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.933733821 (78h34m23.804816856s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.935468164 (78h34m23.806551209s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:25.369562765 (78h34m25.24064581s) [upload=68635]: Next returned nil segment Dec 8 22:03:25.369578857 (78h34m25.240661892s) [upload=68635]: Waiting for error group managing segments... Dec 8 22:03:25.775620953 (78h34m25.646703999s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.798803277 (78h34m25.669886322s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.801642202 (78h34m25.672725237s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.802057232 (78h34m25.673140267s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816132514 (78h34m25.687215559s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816446331 (78h34m25.687529376s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.869308801 (78h34m25.740391846s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.873476507 (78h34m25.744559542s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.907465318 (78h34m25.778548354s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.926737626 (78h34m25.797820671s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.928011191 (78h34m25.799094226s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.930540317 (78h34m25.801623362s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.954520891 (78h34m25.825603926s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001264168 (78h34m25.872347203s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001283154 (78h34m25.872366189s) [seg_pos={0 0}, upload=68635]: Segment reached optimal threshold of 14 pieces. Dec 8 22:03:26.001442167 (78h34m25.872525202s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001448459 (78h34m25.872531494s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001703023 (78h34m25.872786068s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001710758 (78h34m25.872793803s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001867176 (78h34m25.872950231s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001872526 (78h34m25.872955581s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002060293 (78h34m25.873143328s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.002066315 (78h34m25.873149351s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002085732 (78h34m25.873168767s) [seg_pos={0 0}, upload=68635]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:03:26.002156887 (78h34m25.873239922s) [upload=68635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:03:26.002161526 (78h34m25.873244561s) [upload=68635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 22:04:50.380087807 (78h35m50.251170842s) [upload=68635]: Done uploading segments... Dec 8 22:04:50.380137331 (78h35m50.251220376s) [upload=68635]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.600083645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.002120949s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 22:21:34 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.002136868s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 4.734222187s trace: https://jaeger.dev.storj.io/trace/51c602d400b15cea FAIL (2025-12-08 22:21:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.734251534s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.650443279s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:21:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.650471503s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....XXX............................X.X............X.....X..] duration: 3.531916554s trace: https://jaeger.dev.storj.io/trace/5cdb098fa00c6780 FAIL (2025-12-08 22:21:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531965708s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................................X..X.........X..............] duration: 5.666816656s trace: https://jaeger.dev.storj.io/trace/1e577f05655b28db FAIL (2025-12-08 22:21:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.666841042s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X...............X.........................] duration: 5.953907739s trace: https://jaeger.dev.storj.io/trace/72871f7c52601eaf FAIL (2025-12-08 22:20:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917385480 (78h34m23.788468516s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917994611 (78h34m23.789077666s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918445960 (78h34m23.789528995s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918888933 (78h34m23.789971978s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919333861 (78h34m23.790416906s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919736458 (78h34m23.790819503s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920148312 (78h34m23.791231347s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920546540 (78h34m23.791629585s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.921942007 (78h34m23.793025052s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.923813691 (78h34m23.794896736s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.925353112 (78h34m23.796436157s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.927009226 (78h34m23.798092261s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.928697040 (78h34m23.799780085s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.931092261 (78h34m23.802175306s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.932748024 (78h34m23.803831069s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.933733821 (78h34m23.804816856s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.935468164 (78h34m23.806551209s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:25.369562765 (78h34m25.24064581s) [upload=68635]: Next returned nil segment Dec 8 22:03:25.369578857 (78h34m25.240661892s) [upload=68635]: Waiting for error group managing segments... Dec 8 22:03:25.775620953 (78h34m25.646703999s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.798803277 (78h34m25.669886322s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.801642202 (78h34m25.672725237s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.802057232 (78h34m25.673140267s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816132514 (78h34m25.687215559s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816446331 (78h34m25.687529376s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.869308801 (78h34m25.740391846s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.873476507 (78h34m25.744559542s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.907465318 (78h34m25.778548354s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.926737626 (78h34m25.797820671s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.928011191 (78h34m25.799094226s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.930540317 (78h34m25.801623362s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.954520891 (78h34m25.825603926s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001264168 (78h34m25.872347203s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001283154 (78h34m25.872366189s) [seg_pos={0 0}, upload=68635]: Segment reached optimal threshold of 14 pieces. Dec 8 22:03:26.001442167 (78h34m25.872525202s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001448459 (78h34m25.872531494s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001703023 (78h34m25.872786068s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001710758 (78h34m25.872793803s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001867176 (78h34m25.872950231s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001872526 (78h34m25.872955581s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002060293 (78h34m25.873143328s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.002066315 (78h34m25.873149351s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002085732 (78h34m25.873168767s) [seg_pos={0 0}, upload=68635]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:03:26.002156887 (78h34m25.873239922s) [upload=68635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:03:26.002161526 (78h34m25.873244561s) [upload=68635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 22:04:50.380087807 (78h35m50.251170842s) [upload=68635]: Done uploading segments... Dec 8 22:04:50.380137331 (78h35m50.251220376s) [upload=68635]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.953928377s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.977701475s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:20:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.977726092s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 684.089068ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/f605f5110d8504 FAIL (2025-12-08 22:20:47 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 684.111341ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................................X..X.........X.............] duration: 5.867482641s trace: https://jaeger.dev.storj.io/trace/1e577f05655b28db FAIL (2025-12-08 22:20:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.867509742s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X.....XXX............................X.X............X.....X.] duration: 3.467378131s trace: https://jaeger.dev.storj.io/trace/5cdb098fa00c6780 FAIL (2025-12-08 22:20:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46742463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 4.90064087s trace: https://jaeger.dev.storj.io/trace/51c602d400b15cea FAIL (2025-12-08 22:20:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.900668893s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.024062005s trace: https://jaeger.dev.storj.io/trace/1ea7632defe62c27 FAIL (2025-12-08 22:20:27 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.02409075s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 684.509861ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/f605f5110d8504 FAIL (2025-12-08 22:20:17 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 684.538635ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X...............X........................] duration: 5.773108423s trace: https://jaeger.dev.storj.io/trace/72871f7c52601eaf FAIL (2025-12-08 22:20:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== _pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917385480 (78h34m23.788468516s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.917994611 (78h34m23.789077666s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918445960 (78h34m23.789528995s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.918888933 (78h34m23.789971978s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919333861 (78h34m23.790416906s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.919736458 (78h34m23.790819503s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920148312 (78h34m23.791231347s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.920546540 (78h34m23.791629585s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.921942007 (78h34m23.793025052s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.923813691 (78h34m23.794896736s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.925353112 (78h34m23.796436157s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.927009226 (78h34m23.798092261s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.928697040 (78h34m23.799780085s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.931092261 (78h34m23.802175306s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.932748024 (78h34m23.803831069s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.933733821 (78h34m23.804816856s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:23.935468164 (78h34m23.806551209s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Uploading piece... Dec 8 22:03:25.369562765 (78h34m25.24064581s) [upload=68635]: Next returned nil segment Dec 8 22:03:25.369578857 (78h34m25.240661892s) [upload=68635]: Waiting for error group managing segments... Dec 8 22:03:25.775620953 (78h34m25.646703999s) [noise=true , address=64.49.13.26:4951 , piece_id=XKY7QGIG22ASYFNIS6FKWIAYPPKI6GHEACACEIUH6ZKBWTUXMKGQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.798803277 (78h34m25.669886322s) [noise=true , address=145.79.227.7:4307 , piece_id=NPJXZLKRP2DF5DQJVWVVQHHM6MMCANC2V7DKFFPPKJTUMUDTYD6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.801642202 (78h34m25.672725237s) [noise=true , address=145.79.227.33:4876 , piece_id=YFHARIAGOM34XJBLR6DMZ7XX3HZRM5TO3G33KNPMUKZWD4NNTCPA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.802057232 (78h34m25.673140267s) [noise=true , address=145.79.227.13:4831 , piece_id=PA6DXWTKRG5VTOXGZUMXGH4OZ5G33F4OVWYBYYCAJ65NHVTYGQAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816132514 (78h34m25.687215559s) [noise=true , address=193.149.164.143:4694 , piece_id=YB5D6BHSN5N7JFMDC4MOSSYYMVN54LUY6UYF5F3XTUTM6ZBFALCA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.816446331 (78h34m25.687529376s) [noise=true , address=145.79.227.21:4499 , piece_id=ERNE6RSZXWK662FT7VJPQF4SAW24T26HUMO53TOWXOFBYEQZ6J2A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.869308801 (78h34m25.740391846s) [noise=true , address=77.90.31.16:4767 , piece_id=KUBH54G6LYSDL3SY4PHCOT6PAS4BGTF3E77R3ECWJ7TJKWOSQYRA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.873476507 (78h34m25.744559542s) [noise=true , address=5.231.73.4:4006 , piece_id=63SLI6FBO3B3YME3HX4SY2ESSK7LA5PABUL2CEQIUCSTALEVBYDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.907465318 (78h34m25.778548354s) [noise=true , address=77.90.31.14:4677 , piece_id=MZ2RMJMOR45SHFXGGGU22YEQ35BDS452LNCASZ7YB3OCITH6FIDA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.926737626 (78h34m25.797820671s) [noise=true , address=77.90.31.21:4884 , piece_id=TDUSME4Y2CEQJCXP5LKRMS7IZLSVLXRJ32PCA5BOQNHXUA5YBO7A, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.928011191 (78h34m25.799094226s) [noise=true , address=45.91.116.12:4258 , piece_id=DHW72CRHKVV7ZETIQZ4D2TJ66SXMY6QTW67GK7M3C27O3FY2VZKA, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.930540317 (78h34m25.801623362s) [noise=true , address=77.90.31.18:4820 , piece_id=Q5WI2HI5B77A7WS335YDDME3UJFDF6PDGFUIXPVHXFU4RHJT3SZQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:25.954520891 (78h34m25.825603926s) [noise=true , address=77.90.31.17:4801 , piece_id=CIWMVWXXIIMM2K7Q6XU24FCNBUT46AQZ4FUFUCEROI5AJMB5CKAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001264168 (78h34m25.872347203s) [noise=true , address=77.90.31.10:4526 , piece_id=376JCI6NHB3K3T4S7T5OBQL2CPA6CZVUJEDF7HIKZU737MEDQTAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: <nil> Dec 8 22:03:26.001283154 (78h34m25.872366189s) [seg_pos={0 0}, upload=68635]: Segment reached optimal threshold of 14 pieces. Dec 8 22:03:26.001442167 (78h34m25.872525202s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001448459 (78h34m25.872531494s) [noise=true , address=145.79.227.40:4073 , piece_id=PKQLWWXB4PGEJNWBVRODLGTFTKGVDNBWTOMWIWTYDUGADWPD5WAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001703023 (78h34m25.872786068s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123uv49k8LraXJfYQg6KRQthrYs1Egc4c3eZWm5JPvDatFbykm8): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001710758 (78h34m25.872793803s) [noise=true , address=45.91.116.8:4066 , piece_id=GCUGVVC7DLF24CROYKYT4SYEAOFQ7L4RGIDLJH3JE2EO2FAISBYQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.001867176 (78h34m25.872950231s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.001872526 (78h34m25.872955581s) [noise=true , address=77.90.31.13:4637 , piece_id=U6YV74XHJ5Z47MR2XR3ZQY6DLFKYZ6PKESD4MRNUZ22UXNHIAX6Q, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002060293 (78h34m25.873143328s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHCVWGoutBZ3Du4xTCCJGALvrAZwB7cmC1vsRGj5LcwyfW8Qe): protocol: expected piece hash; context canceled; EOF Dec 8 22:03:26.002066315 (78h34m25.873149351s) [noise=true , address=45.91.116.4:4970 , piece_id=GRDPN2OXJLU6DMUPVOETE34QSVWTCAB5IFFO4MQRFSKUGMCVWPAQ, seg_pos={0 0}, upload=68635]: Long tail cancellation due to optimal threshold Dec 8 22:03:26.002085732 (78h34m25.873168767s) [seg_pos={0 0}, upload=68635]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 8 22:03:26.002156887 (78h34m25.873239922s) [upload=68635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 22:03:26.002161526 (78h34m25.873244561s) [upload=68635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 22:04:50.380087807 (78h35m50.251170842s) [upload=68635]: Done uploading segments... Dec 8 22:04:50.380137331 (78h35m50.251220376s) [upload=68635]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.773143129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................................X..X.........X............] duration: 5.963660408s trace: https://jaeger.dev.storj.io/trace/1e577f05655b28db FAIL (2025-12-08 22:20:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.96369227s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.834139359s trace: https://jaeger.dev.storj.io/trace/1aa6e2916420d374 FAIL (2025-12-08 22:19:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.834169746s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 706.268783ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/f605f5110d8504 FAIL (2025-12-08 22:19:48 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 706.295913ms core satellite us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 4.968960002s trace: https://jaeger.dev.storj.io/trace/51c602d400b15cea FAIL (2025-12-08 22:19:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.968979267s core uplink eu1

1
...
11673
11674
11675
11676
11677
...
17319