Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 870158 reports (Page 11594 of 17404)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.X......X...X.XX...X....X...X...X.X..XX...XX.X......X.X.X...] duration: 3.606075863s trace: https://jaeger.dev.storj.io/trace/425fd790d9452b5d FAIL (2025-12-09 15:56:58 UTC) orbiter warning: context deadline exceeded Duration: 3.606107313s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.874349433s trace: https://jaeger.dev.storj.io/trace/2764ea1e2faac6a FAIL (2025-12-09 15:56:52 UTC) orbiter warning: context deadline exceeded Duration: 5.874404988s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 468.163291ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/39a3ed12475253d0 FAIL (2025-12-09 15:56:47 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.17777ms core satellite eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X....X.........................................X...........] duration: 2.425920232s trace: https://jaeger.dev.storj.io/trace/241e9ffbc453b19d FAIL (2025-12-09 15:56:46 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 2.425946612s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...........X............] duration: 5.922686066s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 15:56:43 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.922712856s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..X......X...X.XX...X....X...X...X.X..XX...XX.X......X.X.X..] duration: 3.746460864s trace: https://jaeger.dev.storj.io/trace/425fd790d9452b5d FAIL (2025-12-09 15:56:38 UTC) orbiter warning: context deadline exceeded Duration: 3.746486712s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.......................XX..................X...............] duration: 14.481321082s trace: https://jaeger.dev.storj.io/trace/7e3fae7b586d38d6 FAIL (2025-12-09 15:56:37 UTC) orbiter warning: context deadline exceeded Duration: 14.481345779s edge gateway us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 476.231866ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/39a3ed12475253d0 FAIL (2025-12-09 15:56:11 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 476.257765ms core satellite eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.......................XX..................X..............] duration: 6.065116586s trace: https://jaeger.dev.storj.io/trace/7e3fae7b586d38d6 FAIL (2025-12-09 15:56:09 UTC) orbiter warning: context deadline exceeded Duration: 6.065176289s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.140352461s trace: https://jaeger.dev.storj.io/trace/2764ea1e2faac6a FAIL (2025-12-09 15:56:05 UTC) orbiter warning: context deadline exceeded Duration: 6.140423897s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...X......X...X.XX...X....X...X...X.X..XX...XX.X......X.X.X.] duration: 4.856675988s trace: https://jaeger.dev.storj.io/trace/425fd790d9452b5d FAIL (2025-12-09 15:55:58 UTC) orbiter warning: context deadline exceeded Duration: 4.856704091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...........X...........] duration: 5.862428736s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 15:55:55 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.862452941s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 11.215681236s trace: https://jaeger.dev.storj.io/trace/2764ea1e2faac6a FAIL (2025-12-09 15:55:43 UTC) orbiter warning: context deadline exceeded Duration: 11.215706195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X....X.........................................X..........] duration: 2.295397124s trace: https://jaeger.dev.storj.io/trace/241e9ffbc453b19d FAIL (2025-12-09 15:55:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 2.295418455s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...........X..........] duration: 5.840192266s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 15:55:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.840221432s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.......................XX..................X.............] duration: 7.288222355s trace: https://jaeger.dev.storj.io/trace/7e3fae7b586d38d6 FAIL (2025-12-09 15:55:14 UTC) orbiter warning: context deadline exceeded Duration: 7.288250588s edge gateway us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 469.041385ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/39a3ed12475253d0 FAIL (2025-12-09 15:55:08 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 469.090728ms core satellite eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.......................XX..................X............] duration: 7.278936541s trace: https://jaeger.dev.storj.io/trace/7e3fae7b586d38d6 FAIL (2025-12-09 15:54:57 UTC) orbiter warning: context deadline exceeded Duration: 7.278971127s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [X...X......X...X.XX...X....X...X...X.X..XX...XX.X......X.X.X] duration: 5.002062585s trace: https://jaeger.dev.storj.io/trace/425fd790d9452b5d FAIL (2025-12-09 15:54:54 UTC) orbiter warning: context deadline exceeded Duration: 5.002088864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...X....X.........................................X.........] duration: 2.640608437s trace: https://jaeger.dev.storj.io/trace/241e9ffbc453b19d FAIL (2025-12-09 15:54:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 2.640637634s edge gateway us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 468.034458ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/39a3ed12475253d0 FAIL (2025-12-09 15:54:42 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.077839ms core satellite eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.......................XX..................X...........] duration: 8.351445633s trace: https://jaeger.dev.storj.io/trace/7e3fae7b586d38d6 FAIL (2025-12-09 15:54:40 UTC) orbiter warning: context deadline exceeded Duration: 8.351501401s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 11.809082458s trace: https://jaeger.dev.storj.io/trace/2764ea1e2faac6a FAIL (2025-12-09 15:54:29 UTC) orbiter warning: context deadline exceeded Duration: 11.809111203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...........X.........] duration: 5.892372143s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 15:54:27 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.892405205s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [....X....X.........................................X........] duration: 2.553203938s trace: https://jaeger.dev.storj.io/trace/241e9ffbc453b19d FAIL (2025-12-09 15:54:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 2.553235269s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X....X.......................XX..................X..........] duration: 8.816213044s trace: https://jaeger.dev.storj.io/trace/7e3fae7b586d38d6 FAIL (2025-12-09 15:54:16 UTC) orbiter warning: context deadline exceeded Duration: 8.816239896s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....X....X.........................................X.......] duration: 3.590634888s trace: https://jaeger.dev.storj.io/trace/241e9ffbc453b19d FAIL (2025-12-09 15:54:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 3.590660106s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 9.564351015s trace: https://jaeger.dev.storj.io/trace/2764ea1e2faac6a FAIL (2025-12-09 15:53:59 UTC) orbiter warning: context deadline exceeded Duration: 9.564381563s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 477.901275ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/39a3ed12475253d0 FAIL (2025-12-09 15:53:57 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 477.933475ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XX...X......X...X.XX...X....X...X...X.X..XX...XX.X......X.X.] duration: 5.0003325s trace: https://jaeger.dev.storj.io/trace/4c8c2a8d3e41937b FAIL (2025-12-09 15:53:55 UTC) orbiter warning: context deadline exceeded Duration: 5.000356535s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...........X........] duration: 5.762744855s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 15:53:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.76278432s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 477.328094ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/39a3ed12475253d0 FAIL (2025-12-09 15:53:33 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 477.359052ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.XX...X......X...X.XX...X....X...X...X.X..XX...XX.X......X.X] duration: 5.001053003s trace: https://jaeger.dev.storj.io/trace/4c8c2a8d3e41937b FAIL (2025-12-09 15:53:18 UTC) orbiter warning: context deadline exceeded Duration: 5.001084243s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X....X.......................XX..................X.........] duration: 8.906384608s trace: https://jaeger.dev.storj.io/trace/7e3fae7b586d38d6 FAIL (2025-12-09 15:53:17 UTC) orbiter warning: context deadline exceeded Duration: 8.906413623s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.948418794s trace: https://jaeger.dev.storj.io/trace/2764ea1e2faac6a FAIL (2025-12-09 15:53:07 UTC) orbiter warning: context deadline exceeded Duration: 5.948517503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...........X.......] duration: 5.767306439s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 15:52:57 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.767330396s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [......X....X.........................................X......] duration: 3.534407749s trace: https://jaeger.dev.storj.io/trace/241e9ffbc453b19d FAIL (2025-12-09 15:52:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 3.534426736s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [X.XX...X......X...X.XX...X....X...X...X.X..XX...XX.X......X.] duration: 4.673334855s trace: https://jaeger.dev.storj.io/trace/427d087690b68ef5 FAIL (2025-12-09 15:52:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.673380703s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 467.645615ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/39a3ed12475253d0 FAIL (2025-12-09 15:52:27 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 467.666676ms core satellite eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......X....X.........................................X.....] duration: 2.649077294s trace: https://jaeger.dev.storj.io/trace/241e9ffbc453b19d FAIL (2025-12-09 15:52:14 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 2.64910166s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.322277013s trace: https://jaeger.dev.storj.io/trace/2764ea1e2faac6a FAIL (2025-12-09 15:52:12 UTC) orbiter warning: context deadline exceeded Duration: 6.322324573s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 478.359056ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/39a3ed12475253d0 FAIL (2025-12-09 15:52:11 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 478.389686ms core satellite eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X....X.......................XX..................X........] duration: 10.539130597s trace: https://jaeger.dev.storj.io/trace/7e3fae7b586d38d6 FAIL (2025-12-09 15:52:07 UTC) orbiter warning: context deadline exceeded Duration: 10.539338523s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...........X......] duration: 5.77378843s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 15:52:00 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.773916824s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [........X....X.........................................X....] duration: 3.463333949s trace: https://jaeger.dev.storj.io/trace/241e9ffbc453b19d FAIL (2025-12-09 15:51:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 3.463376139s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..X....X.......................XX..................X.......] duration: 8.744517734s trace: https://jaeger.dev.storj.io/trace/7e3fae7b586d38d6 FAIL (2025-12-09 15:51:37 UTC) orbiter warning: context deadline exceeded Duration: 8.744550425s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [XX.XX...X......X...X.XX...X....X...X...X.X..XX...XX.X......X] duration: 5.001573395s trace: https://jaeger.dev.storj.io/trace/427d087690b68ef5 FAIL (2025-12-09 15:51:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001598583s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.........................................X...........X.....] duration: 5.780847634s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 15:51:31 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.780874034s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 7.785198964s trace: https://jaeger.dev.storj.io/trace/2764ea1e2faac6a FAIL (2025-12-09 15:51:23 UTC) orbiter warning: context deadline exceeded Duration: 7.785241716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.........................................X...........X....] duration: 5.880758166s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 15:51:08 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.880835133s core uplink us-select-1

1
...
11592
11593
11594
11595
11596
...
17404