Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [X.X..XX...XX.X......X.X.X.....X...X....X...X....X......X.XX.] duration: 3.570324288s trace: https://jaeger.dev.storj.io/trace/64f0a308469a3da4 FAIL (2025-12-09 16:20:15 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.570358153s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.146126828s trace: https://jaeger.dev.storj.io/trace/216abfadf4eb4969 FAIL (2025-12-09 16:20:09 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.14615982s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X....................XX............X...XX.......] duration: 18.93966374s trace: https://jaeger.dev.storj.io/trace/6de7fc6276feec11 FAIL (2025-12-09 16:20:07 UTC) orbiter warning: context deadline exceeded Duration: 18.939714316s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...........X............................................] duration: 6.183954029s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 16:20:06 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 6.183989636s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 707.227757ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/18d754292824c46e FAIL (2025-12-09 16:19:55 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 707.262324ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.X.X..XX...XX.X......X.X.X.....X...X....X...X....X......X.XX] duration: 5.00103661s trace: https://jaeger.dev.storj.io/trace/64f0a308469a3da4 FAIL (2025-12-09 16:19:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001060975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.020173363s trace: https://jaeger.dev.storj.io/trace/216abfadf4eb4969 FAIL (2025-12-09 16:19:43 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.02020326s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...........X...........................................] duration: 5.745495925s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 16:19:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.746265139s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................X..........................X..............] duration: 9.840000873s trace: https://jaeger.dev.storj.io/trace/6cc223a408573522 FAIL (2025-12-09 16:19:35 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 9.840081306s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.267169632s trace: https://jaeger.dev.storj.io/trace/216abfadf4eb4969 FAIL (2025-12-09 16:19:28 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.267201002s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X....................XX............X...XX......] duration: 28.787283038s trace: https://jaeger.dev.storj.io/trace/6de7fc6276feec11 FAIL (2025-12-09 16:19:28 UTC) orbiter warning: context deadline exceeded Duration: 28.787310191s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................X..........................X.............] duration: 5.657401791s trace: https://jaeger.dev.storj.io/trace/6cc223a408573522 FAIL (2025-12-09 16:19:16 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.657430956s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.382483721s trace: https://jaeger.dev.storj.io/trace/930e80d185ce30 FAIL (2025-12-09 16:19:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.382516656s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [..X.X..XX...XX.X......X.X.X.....X...X....X...X....X......X.X] duration: 5.001002399s trace: https://jaeger.dev.storj.io/trace/5097c5f9917e8eda FAIL (2025-12-09 16:19:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001028499s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 722.582955ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/18d754292824c46e FAIL (2025-12-09 16:19:02 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 722.614055ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...........X..........................................] duration: 5.920975438s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 16:18:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.921006638s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X....................XX............X...XX.....] duration: 22.675108718s trace: https://jaeger.dev.storj.io/trace/6de7fc6276feec11 FAIL (2025-12-09 16:18:45 UTC) orbiter warning: context deadline exceeded Duration: 22.675121412s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...X.X..XX...XX.X......X.X.X.....X...X....X...X....X......X.] duration: 5.001132595s trace: https://jaeger.dev.storj.io/trace/63a0632e184b85a9 FAIL (2025-12-09 16:18:37 UTC) orbiter warning: context deadline exceeded Duration: 5.00116056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.027748758s trace: https://jaeger.dev.storj.io/trace/216abfadf4eb4969 FAIL (2025-12-09 16:18:22 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.02778206s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............X....................XX............X...XX....] duration: 26.661574714s trace: https://jaeger.dev.storj.io/trace/6de7fc6276feec11 FAIL (2025-12-09 16:18:22 UTC) orbiter warning: context deadline exceeded Duration: 26.661606656s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................X..........................X............] duration: 9.040986302s trace: https://jaeger.dev.storj.io/trace/6cc223a408573522 FAIL (2025-12-09 16:18:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 9.041110077s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...........X.........................................] duration: 5.784831198s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 16:18:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.784859824s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...........X........................................] duration: 5.840643466s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 16:18:00 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.840672632s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.528036238s trace: https://jaeger.dev.storj.io/trace/930e80d185ce30 FAIL (2025-12-09 16:17:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.528062628s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 719.348787ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/18d754292824c46e FAIL (2025-12-09 16:17:48 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 719.379316ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..........................X...........] duration: 8.904754662s trace: https://jaeger.dev.storj.io/trace/6cc223a408573522 FAIL (2025-12-09 16:17:42 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 8.904792013s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X...........X.......................................] duration: 5.716531582s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 16:17:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.71657269s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X...X.X..XX...XX.X......X.X.X.....X...X....X...X....X......X] duration: 5.001493414s trace: https://jaeger.dev.storj.io/trace/63a0632e184b85a9 FAIL (2025-12-09 16:17:34 UTC) orbiter warning: context deadline exceeded Duration: 5.001513924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.07549754s trace: https://jaeger.dev.storj.io/trace/216abfadf4eb4969 FAIL (2025-12-09 16:17:32 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.075530855s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................X....................XX............X...XX...] duration: 24.931602937s trace: https://jaeger.dev.storj.io/trace/6de7fc6276feec11 FAIL (2025-12-09 16:17:21 UTC) orbiter warning: context deadline exceeded Duration: 24.931638565s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.663597375s trace: https://jaeger.dev.storj.io/trace/930e80d185ce30 FAIL (2025-12-09 16:17:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.663635157s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.026964693s trace: https://jaeger.dev.storj.io/trace/216abfadf4eb4969 FAIL (2025-12-09 16:16:55 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.026989119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X...........X......................................] duration: 5.862265959s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 16:16:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.86229266s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.X...X.X..XX...XX.X......X.X.X.....X...X....X...X....X......] duration: 3.613923087s trace: https://jaeger.dev.storj.io/trace/77817e8a9a38e7ff FAIL (2025-12-09 16:16:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.613981779s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 702.920504ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/18d754292824c46e FAIL (2025-12-09 16:16:46 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 702.954158ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.625779084s trace: https://jaeger.dev.storj.io/trace/930e80d185ce30 FAIL (2025-12-09 16:16:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.625808209s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................X..........................X..........] duration: 5.635111969s trace: https://jaeger.dev.storj.io/trace/6cc223a408573522 FAIL (2025-12-09 16:16:35 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.635137448s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................X....................XX............X...XX..] duration: 25.769806399s trace: https://jaeger.dev.storj.io/trace/6de7fc6276feec11 FAIL (2025-12-09 16:16:32 UTC) orbiter warning: context deadline exceeded Duration: 25.769834062s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..X...X.X..XX...XX.X......X.X.X.....X...X....X...X....X.....] duration: 3.684243263s trace: https://jaeger.dev.storj.io/trace/77817e8a9a38e7ff FAIL (2025-12-09 16:16:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.68427351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.242961176s trace: https://jaeger.dev.storj.io/trace/930e80d185ce30 FAIL (2025-12-09 16:16:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.242996342s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X...........X.....................................] duration: 5.726243602s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 16:16:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.726270695s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.998670785s trace: https://jaeger.dev.storj.io/trace/216abfadf4eb4969 FAIL (2025-12-09 16:16:15 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.998691264s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.827869327s trace: https://jaeger.dev.storj.io/trace/930e80d185ce30 FAIL (2025-12-09 16:16:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1m0.827953938s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X....................XX............X...XX.] duration: 22.385654871s trace: https://jaeger.dev.storj.io/trace/6de7fc6276feec11 FAIL (2025-12-09 16:15:54 UTC) orbiter warning: context deadline exceeded Duration: 22.386136197s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..........................X.........] duration: 5.649508283s trace: https://jaeger.dev.storj.io/trace/6cc223a408573522 FAIL (2025-12-09 16:15:43 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.649538582s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.073786153s trace: https://jaeger.dev.storj.io/trace/216abfadf4eb4969 FAIL (2025-12-09 16:15:42 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.073819685s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...X...X.X..XX...XX.X......X.X.X.....X...X....X...X....X....] duration: 3.495955341s trace: https://jaeger.dev.storj.io/trace/77817e8a9a38e7ff FAIL (2025-12-09 16:15:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.495991911s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 716.033495ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/18d754292824c46e FAIL (2025-12-09 16:15:34 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 716.122835ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...........X....................................] duration: 5.788726627s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 16:15:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.78875475s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X...........X...................................] duration: 5.837709441s trace: https://jaeger.dev.storj.io/trace/593b271687016c4f FAIL (2025-12-09 16:15:00 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 15:47:50.798824866 (96h18m50.669907901s) [upload=83937]: Starting upload Dec 9 15:47:50.798921369 (96h18m50.670004404s) [upload=83937]: Done starting upload Dec 9 15:47:50.799010178 (96h18m50.670093213s) [upload=83937]: Uploading segments... Dec 9 15:47:50.799294159 (96h18m50.670377204s) [upload=83937]: Got next segment. Inline: false Dec 9 15:47:50.799363250 (96h18m50.670446295s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 15:47:50.799372037 (96h18m50.670455092s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 15:47:51.056426954 (96h18m50.92751001s) [seg_pos={0 0}, upload=83937]: Begin upload segment... Dec 9 15:47:51.056481187 (96h18m50.927564232s) [seg_pos={0 0}, upload=83937]: Done begin upload segment. Dec 9 15:47:51.056513298 (96h18m50.927596343s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.056576559 (96h18m50.927659594s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057100897 (96h18m50.928183942s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.057596040 (96h18m50.928679075s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058125198 (96h18m50.929208243s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058421582 (96h18m50.929504617s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058696356 (96h18m50.929779391s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.058972071 (96h18m50.930055106s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.059347686 (96h18m50.930430721s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.061182009 (96h18m50.932265044s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.062514146 (96h18m50.933597181s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.064197953 (96h18m50.935280998s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.065677489 (96h18m50.936760534s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.067090440 (96h18m50.938173485s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.068340300 (96h18m50.939423345s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:51.069558109 (96h18m50.940641154s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Uploading piece... Dec 9 15:47:52.703886775 (96h18m52.57496981s) [upload=83937]: Next returned nil segment Dec 9 15:47:52.703900601 (96h18m52.574983636s) [upload=83937]: Waiting for error group managing segments... Dec 9 15:47:53.070939400 (96h18m52.942022445s) [noise=true , address=145.79.227.44:4194 , piece_id=6NCPCDJVVV7F2HY4J6MGR3TD6MOHETXWAUQ7H3FU43AGIBZMBWUA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.074549104 (96h18m52.945632139s) [noise=true , address=145.79.227.39:4014 , piece_id=TNJNQ7DJCHPKOMP2KAW75ZCOMHOBPPWH4XX3WQP5OGZRMCIYHVEA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077053072 (96h18m52.948136107s) [noise=true , address=145.79.227.39:4013 , piece_id=IC654IOYAZRRRKM7G34AXHTQA2QNFBWUP53G3Q4KAXCZOKPI6ZNA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.077574054 (96h18m52.948657099s) [noise=true , address=145.79.227.38:4979 , piece_id=BLERVPIAPSSU7VR26NMIQBRHNUCWLPNYI6E5YD52IBM2JV4CSFEQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.084535460 (96h18m52.955618505s) [noise=true , address=145.79.227.16:4911 , piece_id=7FCNYCTIFXMYEQWKFDU5QVN5XFKSO32RVQTVUDPRIZQCS52I4SBQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.090700220 (96h18m52.961783265s) [noise=true , address=145.79.227.27:4691 , piece_id=7TOCUCNX4ZFERAV2USO5TKQ37JL7E6B3WCYJK4DNPWH4GBC2JHJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.091191014 (96h18m52.962274079s) [noise=true , address=145.79.227.7:4294 , piece_id=NK4AXP6SSK37JGO7OINKL3BH4D23Q5VCGBACQENBPA3HRZ3EMJJA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094302989 (96h18m52.965386025s) [noise=true , address=145.79.227.32:4835 , piece_id=TQB4GDUDDWEYTAUHTDLNVNECRPU5UJWMALOL7KU2S6T4GBR5PM3Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.094537586 (96h18m52.965620622s) [noise=true , address=145.79.227.29:4941 , piece_id=WNUYTTG2LWEKL64R35VQUM7WF4WKLOED4NFFKLY4LPEU27WBMAGA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.115109921 (96h18m52.986192966s) [noise=true , address=145.79.227.42:4122 , piece_id=W5A5G7N2J4775C2TUGUFHC2SR7OSA3AQRMDHSLKTS5K2XC3EHCZQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.118417168 (96h18m52.989500213s) [noise=true , address=145.79.227.43:4158 , piece_id=5FKHKDI3C64XWOUFMNNUSG5PPGK4A4HNX3MWMP4E44CW2BYTBP4A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.120155649 (96h18m52.991238684s) [noise=true , address=145.79.227.23:4547 , piece_id=7LLDJKIMPTDCDLTDARQXO3CZYVCD3OZDD7NOHD7EGBP6XUSPXKGQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123155120 (96h18m52.994238155s) [noise=true , address=77.90.31.14:4661 , piece_id=DGHMAMD7ASLQLLW26A2A4L2KVFD5N2FC5GNJO7UP4W3PFBUHU56Q, seg_pos={0 0}, upload=83937]: Done uploading piece. err: <nil> Dec 9 15:47:53.123168146 (96h18m52.994251181s) [seg_pos={0 0}, upload=83937]: Segment reached optimal threshold of 13 pieces. Dec 9 15:47:53.123387333 (96h18m52.994470368s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123397562 (96h18m52.994480597s) [noise=true , address=77.90.31.21:4879 , piece_id=XDXZ7SACIVSTVQWL5JE4J2XI23UANBGE3RFW7EFL7GLHPPTVPUJQ, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123682554 (96h18m52.994765599s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123691131 (96h18m52.994774176s) [noise=true , address=145.79.227.40:4049 , piece_id=MMRUH72XOW5RIXQTVY75LTIB7OHUD7WUUGXRNTNNWYQQ4AJMWWBA, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123939985 (96h18m52.99502303s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 15:47:53.123949333 (96h18m52.995032378s) [noise=true , address=77.90.31.21:4878 , piece_id=FEVDKBVRLUKDLP77PHAXUUZWE24ARARZ5EOXJCTWZOGY7ME3GA3A, seg_pos={0 0}, upload=83937]: Long tail cancellation due to optimal threshold Dec 9 15:47:53.123987134 (96h18m52.99507017s) [seg_pos={0 0}, upload=83937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 15:47:53.124075293 (96h18m52.995158338s) [upload=83937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 15:47:53.124081745 (96h18m52.99516479s) [upload=83937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 15:48:17.417713882 (96h19m17.288796927s) [upload=83937]: Done uploading segments... Dec 9 15:48:17.417734421 (96h19m17.288817456s) [upload=83937]: Upload finished. err: <nil> Duration: 5.837740019s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)