Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 863607 reports (Page 11721 of 17273)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.110864498s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/687b56ed9d6ebdc3 FAIL (2025-12-08 13:15:39 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.110897251s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.....X..............X...........X.........X.....X.....XX.XX.] duration: 14.726176248s trace: https://jaeger.dev.storj.io/trace/20aa967ba1dbe37f FAIL (2025-12-08 13:15:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.726192779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.X..X...........X......X..X......X....XX.XX.XXXXXXXXXXXXXXXX] duration: 5.001376069s trace: https://jaeger.dev.storj.io/trace/6818b10ff96214cd FAIL (2025-12-08 13:15:20 UTC) orbiter warning: context deadline exceeded Duration: 5.001441333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 7.589312396s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:15:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.589343895s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [......X..............X...........X.........X.....X.....XX.XX] duration: 1m0.000891315s trace: https://jaeger.dev.storj.io/trace/20aa967ba1dbe37f FAIL (2025-12-08 13:15:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000916825s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 474.892339ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:15:07 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 474.925894ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 11.29270649s trace: https://jaeger.dev.storj.io/trace/6af05fea47966629 FAIL (2025-12-08 13:14:55 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:10:14.015116522 (69h41m13.886199567s) [upload=61036]: Starting upload Dec 8 13:10:14.015213327 (69h41m13.886296362s) [upload=61036]: Done starting upload Dec 8 13:10:14.015293729 (69h41m13.886376804s) [upload=61036]: Uploading segments... Dec 8 13:10:14.015421393 (69h41m13.886504438s) [upload=61036]: Got next segment. Inline: false Dec 8 13:10:14.015453564 (69h41m13.886536609s) [upload=61036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:10:14.015458143 (69h41m13.886541188s) [upload=61036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:10:14.263048724 (69h41m14.134131769s) [seg_pos={0 0}, upload=61036]: Begin upload segment... Dec 8 13:10:14.263135330 (69h41m14.134218365s) [seg_pos={0 0}, upload=61036]: Done begin upload segment. Dec 8 13:10:14.263174524 (69h41m14.134257569s) [noise=true , address=77.90.31.15:4699 , piece_id=TZQWWWCUSBWAKYNH75H6BBKM4JWUQIJUAUT3WDUY22AN6VIHMSXQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.263207798 (69h41m14.134290843s) [noise=true , address=145.79.227.43:4157 , piece_id=7FDQLKHIG4ZAWXPECTKJV5C5CKCNVP4SFOTO6OX2X2P4ABI5ZKZQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.263778023 (69h41m14.134861068s) [noise=true , address=145.79.227.22:4510 , piece_id=RTWYMQ5CPQA576VSNDES4H4JADC2FYOTN46YZGKUCURBIVM2V7LA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.264259089 (69h41m14.135342134s) [noise=true , address=77.90.31.13:4627 , piece_id=S2Q7LJDREU3KRSBQTB3V2MTSZFHGAOTUODQBQ2KK34FHJNB3ODOA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.264855755 (69h41m14.1359388s) [noise=true , address=145.79.227.33:4871 , piece_id=TVOPC3636XSIF7SRXRQYOZ7QWMJVIYHMRU6WYOG6VWR4ULVLVZKQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.265440018 (69h41m14.136523053s) [noise=true , address=77.90.31.15:4698 , piece_id=2OZOFM3VACERSSD5Q37RUNESYA5Z7YMEGYL2K5UF525KF7DNL4XA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.265790135 (69h41m14.13687317s) [noise=true , address=77.90.31.19:4843 , piece_id=76TGBWEO5WU7AMJ5X2PMBXLFZB3YUMALJOMGDEMICEFUXR6V7UGA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.266149249 (69h41m14.137232284s) [noise=true , address=145.79.227.23:4546 , piece_id=WX5UGODLH5YXW6K5AQMOH5FYZZMVDHTHBBDTXVVAEIK52IP6HTYA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.267279782 (69h41m14.138362827s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.267442101 (69h41m14.138525186s) [noise=true , address=145.79.227.27:4689 , piece_id=IYOCFLZEKQAN72HA46WIDTNPKWMUYHEEGZETMKAKB6QOBRM6CK3A, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.268658918 (69h41m14.139741953s) [noise=true , address=145.79.227.30:4762 , piece_id=DBCI2WUILFAE5XRM672WKRXOSDLU2MXFYIGNVSGFJK7LOM4ZYDSQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.270535231 (69h41m14.141618276s) [noise=true , address=77.90.31.12:4592 , piece_id=3R5J35R5XAZOE4B466MKD5YASGNV3WUI57LOQ2OCSCEDP5YHI52Q, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.272137752 (69h41m14.143220797s) [noise=true , address=145.79.227.20:4439 , piece_id=BBLZIW5PDVNNHLJQ42EYS37UKCT5ZA37IUPK3GJBBLL2QKUVMDXQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.274342711 (69h41m14.145425756s) [noise=true , address=145.79.227.44:4193 , piece_id=6PKECDCRJWWPFVWV53TSNJPUNU3HOZSVJZJMHKIYIHMFS54N6TYQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.276295780 (69h41m14.147378815s) [noise=true , address=145.79.227.16:4911 , piece_id=2B5VBR3UJKYM2ISCCWED45WG5WVBPEL4Z5F2LEGZJLTITXXDUWJA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.284770187 (69h41m14.155853232s) [noise=true , address=77.90.31.21:4877 , piece_id=QCETBSEYGNHWE27XPPSZQBANF7KDBFZIVLAZDRMMQSSLDO3GLGCA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:16.098535527 (69h41m15.969618573s) [upload=61036]: Next returned nil segment Dec 8 13:10:16.098554383 (69h41m15.969637418s) [upload=61036]: Waiting for error group managing segments... Dec 8 13:10:16.438345353 (69h41m16.309428398s) [noise=true , address=77.90.31.19:4843 , piece_id=76TGBWEO5WU7AMJ5X2PMBXLFZB3YUMALJOMGDEMICEFUXR6V7UGA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:16.482920984 (69h41m16.354004029s) [noise=true , address=77.90.31.12:4592 , piece_id=3R5J35R5XAZOE4B466MKD5YASGNV3WUI57LOQ2OCSCEDP5YHI52Q, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.381455163 (69h41m17.252538208s) [noise=true , address=145.79.227.27:4689 , piece_id=IYOCFLZEKQAN72HA46WIDTNPKWMUYHEEGZETMKAKB6QOBRM6CK3A, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.381660404 (69h41m17.252743449s) [noise=true , address=145.79.227.22:4510 , piece_id=RTWYMQ5CPQA576VSNDES4H4JADC2FYOTN46YZGKUCURBIVM2V7LA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.410802295 (69h41m17.28188534s) [noise=true , address=77.90.31.15:4698 , piece_id=2OZOFM3VACERSSD5Q37RUNESYA5Z7YMEGYL2K5UF525KF7DNL4XA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.414718372 (69h41m17.285801417s) [noise=true , address=145.79.227.23:4546 , piece_id=WX5UGODLH5YXW6K5AQMOH5FYZZMVDHTHBBDTXVVAEIK52IP6HTYA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.433136104 (69h41m17.304219149s) [noise=true , address=145.79.227.43:4157 , piece_id=7FDQLKHIG4ZAWXPECTKJV5C5CKCNVP4SFOTO6OX2X2P4ABI5ZKZQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.448556617 (69h41m17.319639662s) [noise=true , address=145.79.227.44:4193 , piece_id=6PKECDCRJWWPFVWV53TSNJPUNU3HOZSVJZJMHKIYIHMFS54N6TYQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.450453249 (69h41m17.321536294s) [noise=true , address=145.79.227.20:4439 , piece_id=BBLZIW5PDVNNHLJQ42EYS37UKCT5ZA37IUPK3GJBBLL2QKUVMDXQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.457150631 (69h41m17.328233677s) [noise=true , address=145.79.227.33:4871 , piece_id=TVOPC3636XSIF7SRXRQYOZ7QWMJVIYHMRU6WYOG6VWR4ULVLVZKQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.476437257 (69h41m17.347520303s) [noise=true , address=145.79.227.16:4911 , piece_id=2B5VBR3UJKYM2ISCCWED45WG5WVBPEL4Z5F2LEGZJLTITXXDUWJA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.959833941 (69h41m17.830916996s) [noise=true , address=77.90.31.21:4877 , piece_id=QCETBSEYGNHWE27XPPSZQBANF7KDBFZIVLAZDRMMQSSLDO3GLGCA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 13:10:18.258818728 (69h41m18.129901763s) [noise=true , address=77.90.31.15:4699 , piece_id=TZQWWWCUSBWAKYNH75H6BBKM4JWUQIJUAUT3WDUY22AN6VIHMSXQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:10:21.319649531 (69h41m21.190732576s) [noise=true , address=145.79.227.30:4762 , piece_id=DBCI2WUILFAE5XRM672WKRXOSDLU2MXFYIGNVSGFJK7LOM4ZYDSQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:42.550056528 (69h41m42.421139563s) [noise=true , address=77.90.31.13:4627 , piece_id=S2Q7LJDREU3KRSBQTB3V2MTSZFHGAOTUODQBQ2KK34FHJNB3ODOA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:42.550081766 (69h41m42.421164811s) [seg_pos={0 0}, upload=61036]: Segment reached optimal threshold of 13 pieces. Dec 8 13:10:42.550320301 (69h41m42.421403346s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 8 13:10:42.550327524 (69h41m42.421410559s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Long tail cancellation due to optimal threshold Dec 8 13:10:42.550349326 (69h41m42.421432361s) [seg_pos={0 0}, upload=61036]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:10:42.550428026 (69h41m42.421511091s) [upload=61036]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:10:42.550441801 (69h41m42.421524836s) [upload=61036]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:10:42.989209071 (69h41m42.860292116s) [upload=61036]: Done uploading segments... Dec 8 13:10:42.989231784 (69h41m42.860314829s) [upload=61036]: Upload finished. err: <nil> Duration: 11.292727609s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.104997203s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/687b56ed9d6ebdc3 FAIL (2025-12-08 13:14:55 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.105023585s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..X..X...........X......X..X......X....XX.XX.XXXXXXXXXXXXXXX] duration: 5.000863833s trace: https://jaeger.dev.storj.io/trace/23c860d57fb3f14b FAIL (2025-12-08 13:14:37 UTC) orbiter warning: context deadline exceeded Duration: 5.000970107s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X...................................X...X.X.....X....] duration: 28.378074654s trace: https://jaeger.dev.storj.io/trace/7e4a99f6b20554b0 FAIL (2025-12-08 13:14:32 UTC) orbiter warning: context deadline exceeded Duration: 28.37810362s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 11.391801402s trace: https://jaeger.dev.storj.io/trace/6af05fea47966629 FAIL (2025-12-08 13:14:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:10:14.015116522 (69h41m13.886199567s) [upload=61036]: Starting upload Dec 8 13:10:14.015213327 (69h41m13.886296362s) [upload=61036]: Done starting upload Dec 8 13:10:14.015293729 (69h41m13.886376804s) [upload=61036]: Uploading segments... Dec 8 13:10:14.015421393 (69h41m13.886504438s) [upload=61036]: Got next segment. Inline: false Dec 8 13:10:14.015453564 (69h41m13.886536609s) [upload=61036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:10:14.015458143 (69h41m13.886541188s) [upload=61036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:10:14.263048724 (69h41m14.134131769s) [seg_pos={0 0}, upload=61036]: Begin upload segment... Dec 8 13:10:14.263135330 (69h41m14.134218365s) [seg_pos={0 0}, upload=61036]: Done begin upload segment. Dec 8 13:10:14.263174524 (69h41m14.134257569s) [noise=true , address=77.90.31.15:4699 , piece_id=TZQWWWCUSBWAKYNH75H6BBKM4JWUQIJUAUT3WDUY22AN6VIHMSXQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.263207798 (69h41m14.134290843s) [noise=true , address=145.79.227.43:4157 , piece_id=7FDQLKHIG4ZAWXPECTKJV5C5CKCNVP4SFOTO6OX2X2P4ABI5ZKZQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.263778023 (69h41m14.134861068s) [noise=true , address=145.79.227.22:4510 , piece_id=RTWYMQ5CPQA576VSNDES4H4JADC2FYOTN46YZGKUCURBIVM2V7LA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.264259089 (69h41m14.135342134s) [noise=true , address=77.90.31.13:4627 , piece_id=S2Q7LJDREU3KRSBQTB3V2MTSZFHGAOTUODQBQ2KK34FHJNB3ODOA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.264855755 (69h41m14.1359388s) [noise=true , address=145.79.227.33:4871 , piece_id=TVOPC3636XSIF7SRXRQYOZ7QWMJVIYHMRU6WYOG6VWR4ULVLVZKQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.265440018 (69h41m14.136523053s) [noise=true , address=77.90.31.15:4698 , piece_id=2OZOFM3VACERSSD5Q37RUNESYA5Z7YMEGYL2K5UF525KF7DNL4XA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.265790135 (69h41m14.13687317s) [noise=true , address=77.90.31.19:4843 , piece_id=76TGBWEO5WU7AMJ5X2PMBXLFZB3YUMALJOMGDEMICEFUXR6V7UGA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.266149249 (69h41m14.137232284s) [noise=true , address=145.79.227.23:4546 , piece_id=WX5UGODLH5YXW6K5AQMOH5FYZZMVDHTHBBDTXVVAEIK52IP6HTYA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.267279782 (69h41m14.138362827s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.267442101 (69h41m14.138525186s) [noise=true , address=145.79.227.27:4689 , piece_id=IYOCFLZEKQAN72HA46WIDTNPKWMUYHEEGZETMKAKB6QOBRM6CK3A, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.268658918 (69h41m14.139741953s) [noise=true , address=145.79.227.30:4762 , piece_id=DBCI2WUILFAE5XRM672WKRXOSDLU2MXFYIGNVSGFJK7LOM4ZYDSQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.270535231 (69h41m14.141618276s) [noise=true , address=77.90.31.12:4592 , piece_id=3R5J35R5XAZOE4B466MKD5YASGNV3WUI57LOQ2OCSCEDP5YHI52Q, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.272137752 (69h41m14.143220797s) [noise=true , address=145.79.227.20:4439 , piece_id=BBLZIW5PDVNNHLJQ42EYS37UKCT5ZA37IUPK3GJBBLL2QKUVMDXQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.274342711 (69h41m14.145425756s) [noise=true , address=145.79.227.44:4193 , piece_id=6PKECDCRJWWPFVWV53TSNJPUNU3HOZSVJZJMHKIYIHMFS54N6TYQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.276295780 (69h41m14.147378815s) [noise=true , address=145.79.227.16:4911 , piece_id=2B5VBR3UJKYM2ISCCWED45WG5WVBPEL4Z5F2LEGZJLTITXXDUWJA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.284770187 (69h41m14.155853232s) [noise=true , address=77.90.31.21:4877 , piece_id=QCETBSEYGNHWE27XPPSZQBANF7KDBFZIVLAZDRMMQSSLDO3GLGCA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:16.098535527 (69h41m15.969618573s) [upload=61036]: Next returned nil segment Dec 8 13:10:16.098554383 (69h41m15.969637418s) [upload=61036]: Waiting for error group managing segments... Dec 8 13:10:16.438345353 (69h41m16.309428398s) [noise=true , address=77.90.31.19:4843 , piece_id=76TGBWEO5WU7AMJ5X2PMBXLFZB3YUMALJOMGDEMICEFUXR6V7UGA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:16.482920984 (69h41m16.354004029s) [noise=true , address=77.90.31.12:4592 , piece_id=3R5J35R5XAZOE4B466MKD5YASGNV3WUI57LOQ2OCSCEDP5YHI52Q, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.381455163 (69h41m17.252538208s) [noise=true , address=145.79.227.27:4689 , piece_id=IYOCFLZEKQAN72HA46WIDTNPKWMUYHEEGZETMKAKB6QOBRM6CK3A, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.381660404 (69h41m17.252743449s) [noise=true , address=145.79.227.22:4510 , piece_id=RTWYMQ5CPQA576VSNDES4H4JADC2FYOTN46YZGKUCURBIVM2V7LA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.410802295 (69h41m17.28188534s) [noise=true , address=77.90.31.15:4698 , piece_id=2OZOFM3VACERSSD5Q37RUNESYA5Z7YMEGYL2K5UF525KF7DNL4XA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.414718372 (69h41m17.285801417s) [noise=true , address=145.79.227.23:4546 , piece_id=WX5UGODLH5YXW6K5AQMOH5FYZZMVDHTHBBDTXVVAEIK52IP6HTYA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.433136104 (69h41m17.304219149s) [noise=true , address=145.79.227.43:4157 , piece_id=7FDQLKHIG4ZAWXPECTKJV5C5CKCNVP4SFOTO6OX2X2P4ABI5ZKZQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.448556617 (69h41m17.319639662s) [noise=true , address=145.79.227.44:4193 , piece_id=6PKECDCRJWWPFVWV53TSNJPUNU3HOZSVJZJMHKIYIHMFS54N6TYQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.450453249 (69h41m17.321536294s) [noise=true , address=145.79.227.20:4439 , piece_id=BBLZIW5PDVNNHLJQ42EYS37UKCT5ZA37IUPK3GJBBLL2QKUVMDXQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.457150631 (69h41m17.328233677s) [noise=true , address=145.79.227.33:4871 , piece_id=TVOPC3636XSIF7SRXRQYOZ7QWMJVIYHMRU6WYOG6VWR4ULVLVZKQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.476437257 (69h41m17.347520303s) [noise=true , address=145.79.227.16:4911 , piece_id=2B5VBR3UJKYM2ISCCWED45WG5WVBPEL4Z5F2LEGZJLTITXXDUWJA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.959833941 (69h41m17.830916996s) [noise=true , address=77.90.31.21:4877 , piece_id=QCETBSEYGNHWE27XPPSZQBANF7KDBFZIVLAZDRMMQSSLDO3GLGCA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 13:10:18.258818728 (69h41m18.129901763s) [noise=true , address=77.90.31.15:4699 , piece_id=TZQWWWCUSBWAKYNH75H6BBKM4JWUQIJUAUT3WDUY22AN6VIHMSXQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:10:21.319649531 (69h41m21.190732576s) [noise=true , address=145.79.227.30:4762 , piece_id=DBCI2WUILFAE5XRM672WKRXOSDLU2MXFYIGNVSGFJK7LOM4ZYDSQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:42.550056528 (69h41m42.421139563s) [noise=true , address=77.90.31.13:4627 , piece_id=S2Q7LJDREU3KRSBQTB3V2MTSZFHGAOTUODQBQ2KK34FHJNB3ODOA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:42.550081766 (69h41m42.421164811s) [seg_pos={0 0}, upload=61036]: Segment reached optimal threshold of 13 pieces. Dec 8 13:10:42.550320301 (69h41m42.421403346s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 8 13:10:42.550327524 (69h41m42.421410559s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Long tail cancellation due to optimal threshold Dec 8 13:10:42.550349326 (69h41m42.421432361s) [seg_pos={0 0}, upload=61036]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:10:42.550428026 (69h41m42.421511091s) [upload=61036]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:10:42.550441801 (69h41m42.421524836s) [upload=61036]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:10:42.989209071 (69h41m42.860292116s) [upload=61036]: Done uploading segments... Dec 8 13:10:42.989231784 (69h41m42.860314829s) [upload=61036]: Upload finished. err: <nil> Duration: 11.391825477s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 7.978202089s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:14:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.978229251s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 475.614536ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:14:23 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 475.639833ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.......X..............X...........X.........X.....X.....XX.X] duration: 1m0.001015942s trace: https://jaeger.dev.storj.io/trace/1234a947cc0da82a FAIL (2025-12-08 13:14:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001036471s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.072763277s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/687b56ed9d6ebdc3 FAIL (2025-12-08 13:13:54 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.07280139s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [...X..X...........X......X..X......X....XX.XX.XXXXXXXXXXXXXX] duration: 5.001031027s trace: https://jaeger.dev.storj.io/trace/6758467a7aacc1c5 FAIL (2025-12-08 13:13:53 UTC) orbiter warning: context deadline exceeded Duration: 5.001052918s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 474.990978ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:13:35 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 475.015965ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 7.400493492s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:13:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.400567152s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [....X..X...........X......X..X......X....XX.XX.XXXXXXXXXXXXX] duration: 5.001401242s trace: https://jaeger.dev.storj.io/trace/65ceeb86bb6ce7aa FAIL (2025-12-08 13:13:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001422983s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.088210641s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/687b56ed9d6ebdc3 FAIL (2025-12-08 13:13:19 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.088241299s core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X...................................X...X.X.....X...] duration: 16.308848909s trace: https://jaeger.dev.storj.io/trace/7e4a99f6b20554b0 FAIL (2025-12-08 13:13:19 UTC) orbiter warning: context deadline exceeded Duration: 16.308879138s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 8.971905862s trace: https://jaeger.dev.storj.io/trace/6af05fea47966629 FAIL (2025-12-08 13:13:16 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:10:14.015116522 (69h41m13.886199567s) [upload=61036]: Starting upload Dec 8 13:10:14.015213327 (69h41m13.886296362s) [upload=61036]: Done starting upload Dec 8 13:10:14.015293729 (69h41m13.886376804s) [upload=61036]: Uploading segments... Dec 8 13:10:14.015421393 (69h41m13.886504438s) [upload=61036]: Got next segment. Inline: false Dec 8 13:10:14.015453564 (69h41m13.886536609s) [upload=61036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:10:14.015458143 (69h41m13.886541188s) [upload=61036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:10:14.263048724 (69h41m14.134131769s) [seg_pos={0 0}, upload=61036]: Begin upload segment... Dec 8 13:10:14.263135330 (69h41m14.134218365s) [seg_pos={0 0}, upload=61036]: Done begin upload segment. Dec 8 13:10:14.263174524 (69h41m14.134257569s) [noise=true , address=77.90.31.15:4699 , piece_id=TZQWWWCUSBWAKYNH75H6BBKM4JWUQIJUAUT3WDUY22AN6VIHMSXQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.263207798 (69h41m14.134290843s) [noise=true , address=145.79.227.43:4157 , piece_id=7FDQLKHIG4ZAWXPECTKJV5C5CKCNVP4SFOTO6OX2X2P4ABI5ZKZQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.263778023 (69h41m14.134861068s) [noise=true , address=145.79.227.22:4510 , piece_id=RTWYMQ5CPQA576VSNDES4H4JADC2FYOTN46YZGKUCURBIVM2V7LA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.264259089 (69h41m14.135342134s) [noise=true , address=77.90.31.13:4627 , piece_id=S2Q7LJDREU3KRSBQTB3V2MTSZFHGAOTUODQBQ2KK34FHJNB3ODOA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.264855755 (69h41m14.1359388s) [noise=true , address=145.79.227.33:4871 , piece_id=TVOPC3636XSIF7SRXRQYOZ7QWMJVIYHMRU6WYOG6VWR4ULVLVZKQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.265440018 (69h41m14.136523053s) [noise=true , address=77.90.31.15:4698 , piece_id=2OZOFM3VACERSSD5Q37RUNESYA5Z7YMEGYL2K5UF525KF7DNL4XA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.265790135 (69h41m14.13687317s) [noise=true , address=77.90.31.19:4843 , piece_id=76TGBWEO5WU7AMJ5X2PMBXLFZB3YUMALJOMGDEMICEFUXR6V7UGA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.266149249 (69h41m14.137232284s) [noise=true , address=145.79.227.23:4546 , piece_id=WX5UGODLH5YXW6K5AQMOH5FYZZMVDHTHBBDTXVVAEIK52IP6HTYA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.267279782 (69h41m14.138362827s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.267442101 (69h41m14.138525186s) [noise=true , address=145.79.227.27:4689 , piece_id=IYOCFLZEKQAN72HA46WIDTNPKWMUYHEEGZETMKAKB6QOBRM6CK3A, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.268658918 (69h41m14.139741953s) [noise=true , address=145.79.227.30:4762 , piece_id=DBCI2WUILFAE5XRM672WKRXOSDLU2MXFYIGNVSGFJK7LOM4ZYDSQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.270535231 (69h41m14.141618276s) [noise=true , address=77.90.31.12:4592 , piece_id=3R5J35R5XAZOE4B466MKD5YASGNV3WUI57LOQ2OCSCEDP5YHI52Q, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.272137752 (69h41m14.143220797s) [noise=true , address=145.79.227.20:4439 , piece_id=BBLZIW5PDVNNHLJQ42EYS37UKCT5ZA37IUPK3GJBBLL2QKUVMDXQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.274342711 (69h41m14.145425756s) [noise=true , address=145.79.227.44:4193 , piece_id=6PKECDCRJWWPFVWV53TSNJPUNU3HOZSVJZJMHKIYIHMFS54N6TYQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.276295780 (69h41m14.147378815s) [noise=true , address=145.79.227.16:4911 , piece_id=2B5VBR3UJKYM2ISCCWED45WG5WVBPEL4Z5F2LEGZJLTITXXDUWJA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.284770187 (69h41m14.155853232s) [noise=true , address=77.90.31.21:4877 , piece_id=QCETBSEYGNHWE27XPPSZQBANF7KDBFZIVLAZDRMMQSSLDO3GLGCA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:16.098535527 (69h41m15.969618573s) [upload=61036]: Next returned nil segment Dec 8 13:10:16.098554383 (69h41m15.969637418s) [upload=61036]: Waiting for error group managing segments... Dec 8 13:10:16.438345353 (69h41m16.309428398s) [noise=true , address=77.90.31.19:4843 , piece_id=76TGBWEO5WU7AMJ5X2PMBXLFZB3YUMALJOMGDEMICEFUXR6V7UGA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:16.482920984 (69h41m16.354004029s) [noise=true , address=77.90.31.12:4592 , piece_id=3R5J35R5XAZOE4B466MKD5YASGNV3WUI57LOQ2OCSCEDP5YHI52Q, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.381455163 (69h41m17.252538208s) [noise=true , address=145.79.227.27:4689 , piece_id=IYOCFLZEKQAN72HA46WIDTNPKWMUYHEEGZETMKAKB6QOBRM6CK3A, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.381660404 (69h41m17.252743449s) [noise=true , address=145.79.227.22:4510 , piece_id=RTWYMQ5CPQA576VSNDES4H4JADC2FYOTN46YZGKUCURBIVM2V7LA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.410802295 (69h41m17.28188534s) [noise=true , address=77.90.31.15:4698 , piece_id=2OZOFM3VACERSSD5Q37RUNESYA5Z7YMEGYL2K5UF525KF7DNL4XA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.414718372 (69h41m17.285801417s) [noise=true , address=145.79.227.23:4546 , piece_id=WX5UGODLH5YXW6K5AQMOH5FYZZMVDHTHBBDTXVVAEIK52IP6HTYA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.433136104 (69h41m17.304219149s) [noise=true , address=145.79.227.43:4157 , piece_id=7FDQLKHIG4ZAWXPECTKJV5C5CKCNVP4SFOTO6OX2X2P4ABI5ZKZQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.448556617 (69h41m17.319639662s) [noise=true , address=145.79.227.44:4193 , piece_id=6PKECDCRJWWPFVWV53TSNJPUNU3HOZSVJZJMHKIYIHMFS54N6TYQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.450453249 (69h41m17.321536294s) [noise=true , address=145.79.227.20:4439 , piece_id=BBLZIW5PDVNNHLJQ42EYS37UKCT5ZA37IUPK3GJBBLL2QKUVMDXQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.457150631 (69h41m17.328233677s) [noise=true , address=145.79.227.33:4871 , piece_id=TVOPC3636XSIF7SRXRQYOZ7QWMJVIYHMRU6WYOG6VWR4ULVLVZKQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.476437257 (69h41m17.347520303s) [noise=true , address=145.79.227.16:4911 , piece_id=2B5VBR3UJKYM2ISCCWED45WG5WVBPEL4Z5F2LEGZJLTITXXDUWJA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.959833941 (69h41m17.830916996s) [noise=true , address=77.90.31.21:4877 , piece_id=QCETBSEYGNHWE27XPPSZQBANF7KDBFZIVLAZDRMMQSSLDO3GLGCA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 13:10:18.258818728 (69h41m18.129901763s) [noise=true , address=77.90.31.15:4699 , piece_id=TZQWWWCUSBWAKYNH75H6BBKM4JWUQIJUAUT3WDUY22AN6VIHMSXQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:10:21.319649531 (69h41m21.190732576s) [noise=true , address=145.79.227.30:4762 , piece_id=DBCI2WUILFAE5XRM672WKRXOSDLU2MXFYIGNVSGFJK7LOM4ZYDSQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:42.550056528 (69h41m42.421139563s) [noise=true , address=77.90.31.13:4627 , piece_id=S2Q7LJDREU3KRSBQTB3V2MTSZFHGAOTUODQBQ2KK34FHJNB3ODOA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:42.550081766 (69h41m42.421164811s) [seg_pos={0 0}, upload=61036]: Segment reached optimal threshold of 13 pieces. Dec 8 13:10:42.550320301 (69h41m42.421403346s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 8 13:10:42.550327524 (69h41m42.421410559s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Long tail cancellation due to optimal threshold Dec 8 13:10:42.550349326 (69h41m42.421432361s) [seg_pos={0 0}, upload=61036]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:10:42.550428026 (69h41m42.421511091s) [upload=61036]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:10:42.550441801 (69h41m42.421524836s) [upload=61036]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:10:42.989209071 (69h41m42.860292116s) [upload=61036]: Done uploading segments... Dec 8 13:10:42.989231784 (69h41m42.860314829s) [upload=61036]: Upload finished. err: <nil> Duration: 8.971947141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 9.46089865s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:13:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 9.460930492s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [........X..............X...........X.........X.....X.....XX.] duration: 13.919486753s trace: https://jaeger.dev.storj.io/trace/2ba1579e1077564f FAIL (2025-12-08 13:12:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.919502001s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X...................................X...X.X.....X..] duration: 25.576242422s trace: https://jaeger.dev.storj.io/trace/7e4a99f6b20554b0 FAIL (2025-12-08 13:12:46 UTC) orbiter warning: context deadline exceeded Duration: 25.57627255s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.....X..X...........X......X..X......X....XX.XX.XXXXXXXXXXXX] duration: 5.00122931s trace: https://jaeger.dev.storj.io/trace/15cff55d1643325a FAIL (2025-12-08 13:12:37 UTC) orbiter warning: context deadline exceeded Duration: 5.00127138s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 483.463821ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:12:36 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 483.496763ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.........X..............X...........X.........X.....X.....XX] duration: 1m0.001517729s trace: https://jaeger.dev.storj.io/trace/2ba1579e1077564f FAIL (2025-12-08 13:12:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001532187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 7.244092556s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:12:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.244108538s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 13.409376814s trace: https://jaeger.dev.storj.io/trace/6af05fea47966629 FAIL (2025-12-08 13:12:17 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:10:14.015116522 (69h41m13.886199567s) [upload=61036]: Starting upload Dec 8 13:10:14.015213327 (69h41m13.886296362s) [upload=61036]: Done starting upload Dec 8 13:10:14.015293729 (69h41m13.886376804s) [upload=61036]: Uploading segments... Dec 8 13:10:14.015421393 (69h41m13.886504438s) [upload=61036]: Got next segment. Inline: false Dec 8 13:10:14.015453564 (69h41m13.886536609s) [upload=61036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:10:14.015458143 (69h41m13.886541188s) [upload=61036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:10:14.263048724 (69h41m14.134131769s) [seg_pos={0 0}, upload=61036]: Begin upload segment... Dec 8 13:10:14.263135330 (69h41m14.134218365s) [seg_pos={0 0}, upload=61036]: Done begin upload segment. Dec 8 13:10:14.263174524 (69h41m14.134257569s) [noise=true , address=77.90.31.15:4699 , piece_id=TZQWWWCUSBWAKYNH75H6BBKM4JWUQIJUAUT3WDUY22AN6VIHMSXQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.263207798 (69h41m14.134290843s) [noise=true , address=145.79.227.43:4157 , piece_id=7FDQLKHIG4ZAWXPECTKJV5C5CKCNVP4SFOTO6OX2X2P4ABI5ZKZQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.263778023 (69h41m14.134861068s) [noise=true , address=145.79.227.22:4510 , piece_id=RTWYMQ5CPQA576VSNDES4H4JADC2FYOTN46YZGKUCURBIVM2V7LA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.264259089 (69h41m14.135342134s) [noise=true , address=77.90.31.13:4627 , piece_id=S2Q7LJDREU3KRSBQTB3V2MTSZFHGAOTUODQBQ2KK34FHJNB3ODOA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.264855755 (69h41m14.1359388s) [noise=true , address=145.79.227.33:4871 , piece_id=TVOPC3636XSIF7SRXRQYOZ7QWMJVIYHMRU6WYOG6VWR4ULVLVZKQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.265440018 (69h41m14.136523053s) [noise=true , address=77.90.31.15:4698 , piece_id=2OZOFM3VACERSSD5Q37RUNESYA5Z7YMEGYL2K5UF525KF7DNL4XA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.265790135 (69h41m14.13687317s) [noise=true , address=77.90.31.19:4843 , piece_id=76TGBWEO5WU7AMJ5X2PMBXLFZB3YUMALJOMGDEMICEFUXR6V7UGA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.266149249 (69h41m14.137232284s) [noise=true , address=145.79.227.23:4546 , piece_id=WX5UGODLH5YXW6K5AQMOH5FYZZMVDHTHBBDTXVVAEIK52IP6HTYA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.267279782 (69h41m14.138362827s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.267442101 (69h41m14.138525186s) [noise=true , address=145.79.227.27:4689 , piece_id=IYOCFLZEKQAN72HA46WIDTNPKWMUYHEEGZETMKAKB6QOBRM6CK3A, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.268658918 (69h41m14.139741953s) [noise=true , address=145.79.227.30:4762 , piece_id=DBCI2WUILFAE5XRM672WKRXOSDLU2MXFYIGNVSGFJK7LOM4ZYDSQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.270535231 (69h41m14.141618276s) [noise=true , address=77.90.31.12:4592 , piece_id=3R5J35R5XAZOE4B466MKD5YASGNV3WUI57LOQ2OCSCEDP5YHI52Q, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.272137752 (69h41m14.143220797s) [noise=true , address=145.79.227.20:4439 , piece_id=BBLZIW5PDVNNHLJQ42EYS37UKCT5ZA37IUPK3GJBBLL2QKUVMDXQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.274342711 (69h41m14.145425756s) [noise=true , address=145.79.227.44:4193 , piece_id=6PKECDCRJWWPFVWV53TSNJPUNU3HOZSVJZJMHKIYIHMFS54N6TYQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.276295780 (69h41m14.147378815s) [noise=true , address=145.79.227.16:4911 , piece_id=2B5VBR3UJKYM2ISCCWED45WG5WVBPEL4Z5F2LEGZJLTITXXDUWJA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.284770187 (69h41m14.155853232s) [noise=true , address=77.90.31.21:4877 , piece_id=QCETBSEYGNHWE27XPPSZQBANF7KDBFZIVLAZDRMMQSSLDO3GLGCA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:16.098535527 (69h41m15.969618573s) [upload=61036]: Next returned nil segment Dec 8 13:10:16.098554383 (69h41m15.969637418s) [upload=61036]: Waiting for error group managing segments... Dec 8 13:10:16.438345353 (69h41m16.309428398s) [noise=true , address=77.90.31.19:4843 , piece_id=76TGBWEO5WU7AMJ5X2PMBXLFZB3YUMALJOMGDEMICEFUXR6V7UGA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:16.482920984 (69h41m16.354004029s) [noise=true , address=77.90.31.12:4592 , piece_id=3R5J35R5XAZOE4B466MKD5YASGNV3WUI57LOQ2OCSCEDP5YHI52Q, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.381455163 (69h41m17.252538208s) [noise=true , address=145.79.227.27:4689 , piece_id=IYOCFLZEKQAN72HA46WIDTNPKWMUYHEEGZETMKAKB6QOBRM6CK3A, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.381660404 (69h41m17.252743449s) [noise=true , address=145.79.227.22:4510 , piece_id=RTWYMQ5CPQA576VSNDES4H4JADC2FYOTN46YZGKUCURBIVM2V7LA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.410802295 (69h41m17.28188534s) [noise=true , address=77.90.31.15:4698 , piece_id=2OZOFM3VACERSSD5Q37RUNESYA5Z7YMEGYL2K5UF525KF7DNL4XA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.414718372 (69h41m17.285801417s) [noise=true , address=145.79.227.23:4546 , piece_id=WX5UGODLH5YXW6K5AQMOH5FYZZMVDHTHBBDTXVVAEIK52IP6HTYA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.433136104 (69h41m17.304219149s) [noise=true , address=145.79.227.43:4157 , piece_id=7FDQLKHIG4ZAWXPECTKJV5C5CKCNVP4SFOTO6OX2X2P4ABI5ZKZQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.448556617 (69h41m17.319639662s) [noise=true , address=145.79.227.44:4193 , piece_id=6PKECDCRJWWPFVWV53TSNJPUNU3HOZSVJZJMHKIYIHMFS54N6TYQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.450453249 (69h41m17.321536294s) [noise=true , address=145.79.227.20:4439 , piece_id=BBLZIW5PDVNNHLJQ42EYS37UKCT5ZA37IUPK3GJBBLL2QKUVMDXQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.457150631 (69h41m17.328233677s) [noise=true , address=145.79.227.33:4871 , piece_id=TVOPC3636XSIF7SRXRQYOZ7QWMJVIYHMRU6WYOG6VWR4ULVLVZKQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.476437257 (69h41m17.347520303s) [noise=true , address=145.79.227.16:4911 , piece_id=2B5VBR3UJKYM2ISCCWED45WG5WVBPEL4Z5F2LEGZJLTITXXDUWJA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.959833941 (69h41m17.830916996s) [noise=true , address=77.90.31.21:4877 , piece_id=QCETBSEYGNHWE27XPPSZQBANF7KDBFZIVLAZDRMMQSSLDO3GLGCA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 13:10:18.258818728 (69h41m18.129901763s) [noise=true , address=77.90.31.15:4699 , piece_id=TZQWWWCUSBWAKYNH75H6BBKM4JWUQIJUAUT3WDUY22AN6VIHMSXQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:10:21.319649531 (69h41m21.190732576s) [noise=true , address=145.79.227.30:4762 , piece_id=DBCI2WUILFAE5XRM672WKRXOSDLU2MXFYIGNVSGFJK7LOM4ZYDSQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:42.550056528 (69h41m42.421139563s) [noise=true , address=77.90.31.13:4627 , piece_id=S2Q7LJDREU3KRSBQTB3V2MTSZFHGAOTUODQBQ2KK34FHJNB3ODOA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:42.550081766 (69h41m42.421164811s) [seg_pos={0 0}, upload=61036]: Segment reached optimal threshold of 13 pieces. Dec 8 13:10:42.550320301 (69h41m42.421403346s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 8 13:10:42.550327524 (69h41m42.421410559s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Long tail cancellation due to optimal threshold Dec 8 13:10:42.550349326 (69h41m42.421432361s) [seg_pos={0 0}, upload=61036]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:10:42.550428026 (69h41m42.421511091s) [upload=61036]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:10:42.550441801 (69h41m42.421524836s) [upload=61036]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:10:42.989209071 (69h41m42.860292116s) [upload=61036]: Done uploading segments... Dec 8 13:10:42.989231784 (69h41m42.860314829s) [upload=61036]: Upload finished. err: <nil> Duration: 13.40941128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 13m36.600977173s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:12:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 13m36.600998784s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.081428627s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/687b56ed9d6ebdc3 FAIL (2025-12-08 13:12:16 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.081455067s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X.....X..X...........X......X..X......X....XX.XX.XXXXXXXXXXX] duration: 5.001755081s trace: https://jaeger.dev.storj.io/trace/c219102124a284f FAIL (2025-12-08 13:12:12 UTC) orbiter warning: context deadline exceeded Duration: 5.001778778s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X...................................X...X.X.....X.] duration: 22.528589461s trace: https://jaeger.dev.storj.io/trace/7e4a99f6b20554b0 FAIL (2025-12-08 13:11:58 UTC) orbiter warning: context deadline exceeded Duration: 22.528616783s edge gateway us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 469.107334ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:11:39 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 469.144125ms core satellite eu1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.093262742s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/687b56ed9d6ebdc3 FAIL (2025-12-08 13:11:36 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.093284853s core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..........X..............X...........X.........X.....X.....X] duration: 1m0.001198201s trace: https://jaeger.dev.storj.io/trace/7dbfe2053d723ad9 FAIL (2025-12-08 13:11:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001222948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 9.8172405s trace: https://jaeger.dev.storj.io/trace/6af05fea47966629 FAIL (2025-12-08 13:11:32 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:10:14.015116522 (69h41m13.886199567s) [upload=61036]: Starting upload Dec 8 13:10:14.015213327 (69h41m13.886296362s) [upload=61036]: Done starting upload Dec 8 13:10:14.015293729 (69h41m13.886376804s) [upload=61036]: Uploading segments... Dec 8 13:10:14.015421393 (69h41m13.886504438s) [upload=61036]: Got next segment. Inline: false Dec 8 13:10:14.015453564 (69h41m13.886536609s) [upload=61036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:10:14.015458143 (69h41m13.886541188s) [upload=61036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:10:14.263048724 (69h41m14.134131769s) [seg_pos={0 0}, upload=61036]: Begin upload segment... Dec 8 13:10:14.263135330 (69h41m14.134218365s) [seg_pos={0 0}, upload=61036]: Done begin upload segment. Dec 8 13:10:14.263174524 (69h41m14.134257569s) [noise=true , address=77.90.31.15:4699 , piece_id=TZQWWWCUSBWAKYNH75H6BBKM4JWUQIJUAUT3WDUY22AN6VIHMSXQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.263207798 (69h41m14.134290843s) [noise=true , address=145.79.227.43:4157 , piece_id=7FDQLKHIG4ZAWXPECTKJV5C5CKCNVP4SFOTO6OX2X2P4ABI5ZKZQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.263778023 (69h41m14.134861068s) [noise=true , address=145.79.227.22:4510 , piece_id=RTWYMQ5CPQA576VSNDES4H4JADC2FYOTN46YZGKUCURBIVM2V7LA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.264259089 (69h41m14.135342134s) [noise=true , address=77.90.31.13:4627 , piece_id=S2Q7LJDREU3KRSBQTB3V2MTSZFHGAOTUODQBQ2KK34FHJNB3ODOA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.264855755 (69h41m14.1359388s) [noise=true , address=145.79.227.33:4871 , piece_id=TVOPC3636XSIF7SRXRQYOZ7QWMJVIYHMRU6WYOG6VWR4ULVLVZKQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.265440018 (69h41m14.136523053s) [noise=true , address=77.90.31.15:4698 , piece_id=2OZOFM3VACERSSD5Q37RUNESYA5Z7YMEGYL2K5UF525KF7DNL4XA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.265790135 (69h41m14.13687317s) [noise=true , address=77.90.31.19:4843 , piece_id=76TGBWEO5WU7AMJ5X2PMBXLFZB3YUMALJOMGDEMICEFUXR6V7UGA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.266149249 (69h41m14.137232284s) [noise=true , address=145.79.227.23:4546 , piece_id=WX5UGODLH5YXW6K5AQMOH5FYZZMVDHTHBBDTXVVAEIK52IP6HTYA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.267279782 (69h41m14.138362827s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.267442101 (69h41m14.138525186s) [noise=true , address=145.79.227.27:4689 , piece_id=IYOCFLZEKQAN72HA46WIDTNPKWMUYHEEGZETMKAKB6QOBRM6CK3A, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.268658918 (69h41m14.139741953s) [noise=true , address=145.79.227.30:4762 , piece_id=DBCI2WUILFAE5XRM672WKRXOSDLU2MXFYIGNVSGFJK7LOM4ZYDSQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.270535231 (69h41m14.141618276s) [noise=true , address=77.90.31.12:4592 , piece_id=3R5J35R5XAZOE4B466MKD5YASGNV3WUI57LOQ2OCSCEDP5YHI52Q, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.272137752 (69h41m14.143220797s) [noise=true , address=145.79.227.20:4439 , piece_id=BBLZIW5PDVNNHLJQ42EYS37UKCT5ZA37IUPK3GJBBLL2QKUVMDXQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.274342711 (69h41m14.145425756s) [noise=true , address=145.79.227.44:4193 , piece_id=6PKECDCRJWWPFVWV53TSNJPUNU3HOZSVJZJMHKIYIHMFS54N6TYQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.276295780 (69h41m14.147378815s) [noise=true , address=145.79.227.16:4911 , piece_id=2B5VBR3UJKYM2ISCCWED45WG5WVBPEL4Z5F2LEGZJLTITXXDUWJA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.284770187 (69h41m14.155853232s) [noise=true , address=77.90.31.21:4877 , piece_id=QCETBSEYGNHWE27XPPSZQBANF7KDBFZIVLAZDRMMQSSLDO3GLGCA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:16.098535527 (69h41m15.969618573s) [upload=61036]: Next returned nil segment Dec 8 13:10:16.098554383 (69h41m15.969637418s) [upload=61036]: Waiting for error group managing segments... Dec 8 13:10:16.438345353 (69h41m16.309428398s) [noise=true , address=77.90.31.19:4843 , piece_id=76TGBWEO5WU7AMJ5X2PMBXLFZB3YUMALJOMGDEMICEFUXR6V7UGA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:16.482920984 (69h41m16.354004029s) [noise=true , address=77.90.31.12:4592 , piece_id=3R5J35R5XAZOE4B466MKD5YASGNV3WUI57LOQ2OCSCEDP5YHI52Q, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.381455163 (69h41m17.252538208s) [noise=true , address=145.79.227.27:4689 , piece_id=IYOCFLZEKQAN72HA46WIDTNPKWMUYHEEGZETMKAKB6QOBRM6CK3A, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.381660404 (69h41m17.252743449s) [noise=true , address=145.79.227.22:4510 , piece_id=RTWYMQ5CPQA576VSNDES4H4JADC2FYOTN46YZGKUCURBIVM2V7LA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.410802295 (69h41m17.28188534s) [noise=true , address=77.90.31.15:4698 , piece_id=2OZOFM3VACERSSD5Q37RUNESYA5Z7YMEGYL2K5UF525KF7DNL4XA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.414718372 (69h41m17.285801417s) [noise=true , address=145.79.227.23:4546 , piece_id=WX5UGODLH5YXW6K5AQMOH5FYZZMVDHTHBBDTXVVAEIK52IP6HTYA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.433136104 (69h41m17.304219149s) [noise=true , address=145.79.227.43:4157 , piece_id=7FDQLKHIG4ZAWXPECTKJV5C5CKCNVP4SFOTO6OX2X2P4ABI5ZKZQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.448556617 (69h41m17.319639662s) [noise=true , address=145.79.227.44:4193 , piece_id=6PKECDCRJWWPFVWV53TSNJPUNU3HOZSVJZJMHKIYIHMFS54N6TYQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.450453249 (69h41m17.321536294s) [noise=true , address=145.79.227.20:4439 , piece_id=BBLZIW5PDVNNHLJQ42EYS37UKCT5ZA37IUPK3GJBBLL2QKUVMDXQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.457150631 (69h41m17.328233677s) [noise=true , address=145.79.227.33:4871 , piece_id=TVOPC3636XSIF7SRXRQYOZ7QWMJVIYHMRU6WYOG6VWR4ULVLVZKQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.476437257 (69h41m17.347520303s) [noise=true , address=145.79.227.16:4911 , piece_id=2B5VBR3UJKYM2ISCCWED45WG5WVBPEL4Z5F2LEGZJLTITXXDUWJA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.959833941 (69h41m17.830916996s) [noise=true , address=77.90.31.21:4877 , piece_id=QCETBSEYGNHWE27XPPSZQBANF7KDBFZIVLAZDRMMQSSLDO3GLGCA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 13:10:18.258818728 (69h41m18.129901763s) [noise=true , address=77.90.31.15:4699 , piece_id=TZQWWWCUSBWAKYNH75H6BBKM4JWUQIJUAUT3WDUY22AN6VIHMSXQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:10:21.319649531 (69h41m21.190732576s) [noise=true , address=145.79.227.30:4762 , piece_id=DBCI2WUILFAE5XRM672WKRXOSDLU2MXFYIGNVSGFJK7LOM4ZYDSQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:42.550056528 (69h41m42.421139563s) [noise=true , address=77.90.31.13:4627 , piece_id=S2Q7LJDREU3KRSBQTB3V2MTSZFHGAOTUODQBQ2KK34FHJNB3ODOA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:42.550081766 (69h41m42.421164811s) [seg_pos={0 0}, upload=61036]: Segment reached optimal threshold of 13 pieces. Dec 8 13:10:42.550320301 (69h41m42.421403346s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 8 13:10:42.550327524 (69h41m42.421410559s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Long tail cancellation due to optimal threshold Dec 8 13:10:42.550349326 (69h41m42.421432361s) [seg_pos={0 0}, upload=61036]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:10:42.550428026 (69h41m42.421511091s) [upload=61036]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:10:42.550441801 (69h41m42.421524836s) [upload=61036]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:10:42.989209071 (69h41m42.860292116s) [upload=61036]: Done uploading segments... Dec 8 13:10:42.989231784 (69h41m42.860314829s) [upload=61036]: Upload finished. err: <nil> Duration: 9.817268233s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X...................................X...X.X.....X] duration: 30.001050299s trace: https://jaeger.dev.storj.io/trace/7e4a99f6b20554b0 FAIL (2025-12-08 13:11:26 UTC) orbiter warning: context deadline exceeded Duration: 30.001074465s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [XX.....X..X...........X......X..X......X....XX.XX.XXXXXXXXXX] duration: 5.002249923s trace: https://jaeger.dev.storj.io/trace/647d2de4c409ac10 FAIL (2025-12-08 13:11:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002289359s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.XX.....X..X...........X......X..X......X....XX.XX.XXXXXXXXX] duration: 5.001249291s trace: https://jaeger.dev.storj.io/trace/6ffcbc4ae97dd368 FAIL (2025-12-08 13:11:10 UTC) orbiter warning: context deadline exceeded Duration: 5.001272936s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 13.862270867s trace: https://jaeger.dev.storj.io/trace/6af05fea47966629 FAIL (2025-12-08 13:11:08 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:10:14.015116522 (69h41m13.886199567s) [upload=61036]: Starting upload Dec 8 13:10:14.015213327 (69h41m13.886296362s) [upload=61036]: Done starting upload Dec 8 13:10:14.015293729 (69h41m13.886376804s) [upload=61036]: Uploading segments... Dec 8 13:10:14.015421393 (69h41m13.886504438s) [upload=61036]: Got next segment. Inline: false Dec 8 13:10:14.015453564 (69h41m13.886536609s) [upload=61036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:10:14.015458143 (69h41m13.886541188s) [upload=61036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:10:14.263048724 (69h41m14.134131769s) [seg_pos={0 0}, upload=61036]: Begin upload segment... Dec 8 13:10:14.263135330 (69h41m14.134218365s) [seg_pos={0 0}, upload=61036]: Done begin upload segment. Dec 8 13:10:14.263174524 (69h41m14.134257569s) [noise=true , address=77.90.31.15:4699 , piece_id=TZQWWWCUSBWAKYNH75H6BBKM4JWUQIJUAUT3WDUY22AN6VIHMSXQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.263207798 (69h41m14.134290843s) [noise=true , address=145.79.227.43:4157 , piece_id=7FDQLKHIG4ZAWXPECTKJV5C5CKCNVP4SFOTO6OX2X2P4ABI5ZKZQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.263778023 (69h41m14.134861068s) [noise=true , address=145.79.227.22:4510 , piece_id=RTWYMQ5CPQA576VSNDES4H4JADC2FYOTN46YZGKUCURBIVM2V7LA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.264259089 (69h41m14.135342134s) [noise=true , address=77.90.31.13:4627 , piece_id=S2Q7LJDREU3KRSBQTB3V2MTSZFHGAOTUODQBQ2KK34FHJNB3ODOA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.264855755 (69h41m14.1359388s) [noise=true , address=145.79.227.33:4871 , piece_id=TVOPC3636XSIF7SRXRQYOZ7QWMJVIYHMRU6WYOG6VWR4ULVLVZKQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.265440018 (69h41m14.136523053s) [noise=true , address=77.90.31.15:4698 , piece_id=2OZOFM3VACERSSD5Q37RUNESYA5Z7YMEGYL2K5UF525KF7DNL4XA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.265790135 (69h41m14.13687317s) [noise=true , address=77.90.31.19:4843 , piece_id=76TGBWEO5WU7AMJ5X2PMBXLFZB3YUMALJOMGDEMICEFUXR6V7UGA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.266149249 (69h41m14.137232284s) [noise=true , address=145.79.227.23:4546 , piece_id=WX5UGODLH5YXW6K5AQMOH5FYZZMVDHTHBBDTXVVAEIK52IP6HTYA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.267279782 (69h41m14.138362827s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.267442101 (69h41m14.138525186s) [noise=true , address=145.79.227.27:4689 , piece_id=IYOCFLZEKQAN72HA46WIDTNPKWMUYHEEGZETMKAKB6QOBRM6CK3A, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.268658918 (69h41m14.139741953s) [noise=true , address=145.79.227.30:4762 , piece_id=DBCI2WUILFAE5XRM672WKRXOSDLU2MXFYIGNVSGFJK7LOM4ZYDSQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.270535231 (69h41m14.141618276s) [noise=true , address=77.90.31.12:4592 , piece_id=3R5J35R5XAZOE4B466MKD5YASGNV3WUI57LOQ2OCSCEDP5YHI52Q, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.272137752 (69h41m14.143220797s) [noise=true , address=145.79.227.20:4439 , piece_id=BBLZIW5PDVNNHLJQ42EYS37UKCT5ZA37IUPK3GJBBLL2QKUVMDXQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.274342711 (69h41m14.145425756s) [noise=true , address=145.79.227.44:4193 , piece_id=6PKECDCRJWWPFVWV53TSNJPUNU3HOZSVJZJMHKIYIHMFS54N6TYQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.276295780 (69h41m14.147378815s) [noise=true , address=145.79.227.16:4911 , piece_id=2B5VBR3UJKYM2ISCCWED45WG5WVBPEL4Z5F2LEGZJLTITXXDUWJA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.284770187 (69h41m14.155853232s) [noise=true , address=77.90.31.21:4877 , piece_id=QCETBSEYGNHWE27XPPSZQBANF7KDBFZIVLAZDRMMQSSLDO3GLGCA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:16.098535527 (69h41m15.969618573s) [upload=61036]: Next returned nil segment Dec 8 13:10:16.098554383 (69h41m15.969637418s) [upload=61036]: Waiting for error group managing segments... Dec 8 13:10:16.438345353 (69h41m16.309428398s) [noise=true , address=77.90.31.19:4843 , piece_id=76TGBWEO5WU7AMJ5X2PMBXLFZB3YUMALJOMGDEMICEFUXR6V7UGA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:16.482920984 (69h41m16.354004029s) [noise=true , address=77.90.31.12:4592 , piece_id=3R5J35R5XAZOE4B466MKD5YASGNV3WUI57LOQ2OCSCEDP5YHI52Q, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.381455163 (69h41m17.252538208s) [noise=true , address=145.79.227.27:4689 , piece_id=IYOCFLZEKQAN72HA46WIDTNPKWMUYHEEGZETMKAKB6QOBRM6CK3A, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.381660404 (69h41m17.252743449s) [noise=true , address=145.79.227.22:4510 , piece_id=RTWYMQ5CPQA576VSNDES4H4JADC2FYOTN46YZGKUCURBIVM2V7LA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.410802295 (69h41m17.28188534s) [noise=true , address=77.90.31.15:4698 , piece_id=2OZOFM3VACERSSD5Q37RUNESYA5Z7YMEGYL2K5UF525KF7DNL4XA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.414718372 (69h41m17.285801417s) [noise=true , address=145.79.227.23:4546 , piece_id=WX5UGODLH5YXW6K5AQMOH5FYZZMVDHTHBBDTXVVAEIK52IP6HTYA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.433136104 (69h41m17.304219149s) [noise=true , address=145.79.227.43:4157 , piece_id=7FDQLKHIG4ZAWXPECTKJV5C5CKCNVP4SFOTO6OX2X2P4ABI5ZKZQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.448556617 (69h41m17.319639662s) [noise=true , address=145.79.227.44:4193 , piece_id=6PKECDCRJWWPFVWV53TSNJPUNU3HOZSVJZJMHKIYIHMFS54N6TYQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.450453249 (69h41m17.321536294s) [noise=true , address=145.79.227.20:4439 , piece_id=BBLZIW5PDVNNHLJQ42EYS37UKCT5ZA37IUPK3GJBBLL2QKUVMDXQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.457150631 (69h41m17.328233677s) [noise=true , address=145.79.227.33:4871 , piece_id=TVOPC3636XSIF7SRXRQYOZ7QWMJVIYHMRU6WYOG6VWR4ULVLVZKQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.476437257 (69h41m17.347520303s) [noise=true , address=145.79.227.16:4911 , piece_id=2B5VBR3UJKYM2ISCCWED45WG5WVBPEL4Z5F2LEGZJLTITXXDUWJA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.959833941 (69h41m17.830916996s) [noise=true , address=77.90.31.21:4877 , piece_id=QCETBSEYGNHWE27XPPSZQBANF7KDBFZIVLAZDRMMQSSLDO3GLGCA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 13:10:18.258818728 (69h41m18.129901763s) [noise=true , address=77.90.31.15:4699 , piece_id=TZQWWWCUSBWAKYNH75H6BBKM4JWUQIJUAUT3WDUY22AN6VIHMSXQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:10:21.319649531 (69h41m21.190732576s) [noise=true , address=145.79.227.30:4762 , piece_id=DBCI2WUILFAE5XRM672WKRXOSDLU2MXFYIGNVSGFJK7LOM4ZYDSQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:42.550056528 (69h41m42.421139563s) [noise=true , address=77.90.31.13:4627 , piece_id=S2Q7LJDREU3KRSBQTB3V2MTSZFHGAOTUODQBQ2KK34FHJNB3ODOA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:42.550081766 (69h41m42.421164811s) [seg_pos={0 0}, upload=61036]: Segment reached optimal threshold of 13 pieces. Dec 8 13:10:42.550320301 (69h41m42.421403346s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 8 13:10:42.550327524 (69h41m42.421410559s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Long tail cancellation due to optimal threshold Dec 8 13:10:42.550349326 (69h41m42.421432361s) [seg_pos={0 0}, upload=61036]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:10:42.550428026 (69h41m42.421511091s) [upload=61036]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:10:42.550441801 (69h41m42.421524836s) [upload=61036]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:10:42.989209071 (69h41m42.860292116s) [upload=61036]: Done uploading segments... Dec 8 13:10:42.989231784 (69h41m42.860314829s) [upload=61036]: Upload finished. err: <nil> Duration: 13.862299972s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002828648s trace: https://jaeger.dev.storj.io/trace/6af05fea47966629 FAIL (2025-12-08 13:10:43 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:10:14.015116522 (69h41m13.886199567s) [upload=61036]: Starting upload Dec 8 13:10:14.015213327 (69h41m13.886296362s) [upload=61036]: Done starting upload Dec 8 13:10:14.015293729 (69h41m13.886376804s) [upload=61036]: Uploading segments... Dec 8 13:10:14.015421393 (69h41m13.886504438s) [upload=61036]: Got next segment. Inline: false Dec 8 13:10:14.015453564 (69h41m13.886536609s) [upload=61036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:10:14.015458143 (69h41m13.886541188s) [upload=61036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:10:14.263048724 (69h41m14.134131769s) [seg_pos={0 0}, upload=61036]: Begin upload segment... Dec 8 13:10:14.263135330 (69h41m14.134218365s) [seg_pos={0 0}, upload=61036]: Done begin upload segment. Dec 8 13:10:14.263174524 (69h41m14.134257569s) [noise=true , address=77.90.31.15:4699 , piece_id=TZQWWWCUSBWAKYNH75H6BBKM4JWUQIJUAUT3WDUY22AN6VIHMSXQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.263207798 (69h41m14.134290843s) [noise=true , address=145.79.227.43:4157 , piece_id=7FDQLKHIG4ZAWXPECTKJV5C5CKCNVP4SFOTO6OX2X2P4ABI5ZKZQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.263778023 (69h41m14.134861068s) [noise=true , address=145.79.227.22:4510 , piece_id=RTWYMQ5CPQA576VSNDES4H4JADC2FYOTN46YZGKUCURBIVM2V7LA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.264259089 (69h41m14.135342134s) [noise=true , address=77.90.31.13:4627 , piece_id=S2Q7LJDREU3KRSBQTB3V2MTSZFHGAOTUODQBQ2KK34FHJNB3ODOA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.264855755 (69h41m14.1359388s) [noise=true , address=145.79.227.33:4871 , piece_id=TVOPC3636XSIF7SRXRQYOZ7QWMJVIYHMRU6WYOG6VWR4ULVLVZKQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.265440018 (69h41m14.136523053s) [noise=true , address=77.90.31.15:4698 , piece_id=2OZOFM3VACERSSD5Q37RUNESYA5Z7YMEGYL2K5UF525KF7DNL4XA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.265790135 (69h41m14.13687317s) [noise=true , address=77.90.31.19:4843 , piece_id=76TGBWEO5WU7AMJ5X2PMBXLFZB3YUMALJOMGDEMICEFUXR6V7UGA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.266149249 (69h41m14.137232284s) [noise=true , address=145.79.227.23:4546 , piece_id=WX5UGODLH5YXW6K5AQMOH5FYZZMVDHTHBBDTXVVAEIK52IP6HTYA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.267279782 (69h41m14.138362827s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.267442101 (69h41m14.138525186s) [noise=true , address=145.79.227.27:4689 , piece_id=IYOCFLZEKQAN72HA46WIDTNPKWMUYHEEGZETMKAKB6QOBRM6CK3A, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.268658918 (69h41m14.139741953s) [noise=true , address=145.79.227.30:4762 , piece_id=DBCI2WUILFAE5XRM672WKRXOSDLU2MXFYIGNVSGFJK7LOM4ZYDSQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.270535231 (69h41m14.141618276s) [noise=true , address=77.90.31.12:4592 , piece_id=3R5J35R5XAZOE4B466MKD5YASGNV3WUI57LOQ2OCSCEDP5YHI52Q, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.272137752 (69h41m14.143220797s) [noise=true , address=145.79.227.20:4439 , piece_id=BBLZIW5PDVNNHLJQ42EYS37UKCT5ZA37IUPK3GJBBLL2QKUVMDXQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.274342711 (69h41m14.145425756s) [noise=true , address=145.79.227.44:4193 , piece_id=6PKECDCRJWWPFVWV53TSNJPUNU3HOZSVJZJMHKIYIHMFS54N6TYQ, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.276295780 (69h41m14.147378815s) [noise=true , address=145.79.227.16:4911 , piece_id=2B5VBR3UJKYM2ISCCWED45WG5WVBPEL4Z5F2LEGZJLTITXXDUWJA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:14.284770187 (69h41m14.155853232s) [noise=true , address=77.90.31.21:4877 , piece_id=QCETBSEYGNHWE27XPPSZQBANF7KDBFZIVLAZDRMMQSSLDO3GLGCA, seg_pos={0 0}, upload=61036]: Uploading piece... Dec 8 13:10:16.098535527 (69h41m15.969618573s) [upload=61036]: Next returned nil segment Dec 8 13:10:16.098554383 (69h41m15.969637418s) [upload=61036]: Waiting for error group managing segments... Dec 8 13:10:16.438345353 (69h41m16.309428398s) [noise=true , address=77.90.31.19:4843 , piece_id=76TGBWEO5WU7AMJ5X2PMBXLFZB3YUMALJOMGDEMICEFUXR6V7UGA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:16.482920984 (69h41m16.354004029s) [noise=true , address=77.90.31.12:4592 , piece_id=3R5J35R5XAZOE4B466MKD5YASGNV3WUI57LOQ2OCSCEDP5YHI52Q, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.381455163 (69h41m17.252538208s) [noise=true , address=145.79.227.27:4689 , piece_id=IYOCFLZEKQAN72HA46WIDTNPKWMUYHEEGZETMKAKB6QOBRM6CK3A, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.381660404 (69h41m17.252743449s) [noise=true , address=145.79.227.22:4510 , piece_id=RTWYMQ5CPQA576VSNDES4H4JADC2FYOTN46YZGKUCURBIVM2V7LA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.410802295 (69h41m17.28188534s) [noise=true , address=77.90.31.15:4698 , piece_id=2OZOFM3VACERSSD5Q37RUNESYA5Z7YMEGYL2K5UF525KF7DNL4XA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.414718372 (69h41m17.285801417s) [noise=true , address=145.79.227.23:4546 , piece_id=WX5UGODLH5YXW6K5AQMOH5FYZZMVDHTHBBDTXVVAEIK52IP6HTYA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.433136104 (69h41m17.304219149s) [noise=true , address=145.79.227.43:4157 , piece_id=7FDQLKHIG4ZAWXPECTKJV5C5CKCNVP4SFOTO6OX2X2P4ABI5ZKZQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.448556617 (69h41m17.319639662s) [noise=true , address=145.79.227.44:4193 , piece_id=6PKECDCRJWWPFVWV53TSNJPUNU3HOZSVJZJMHKIYIHMFS54N6TYQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.450453249 (69h41m17.321536294s) [noise=true , address=145.79.227.20:4439 , piece_id=BBLZIW5PDVNNHLJQ42EYS37UKCT5ZA37IUPK3GJBBLL2QKUVMDXQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.457150631 (69h41m17.328233677s) [noise=true , address=145.79.227.33:4871 , piece_id=TVOPC3636XSIF7SRXRQYOZ7QWMJVIYHMRU6WYOG6VWR4ULVLVZKQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.476437257 (69h41m17.347520303s) [noise=true , address=145.79.227.16:4911 , piece_id=2B5VBR3UJKYM2ISCCWED45WG5WVBPEL4Z5F2LEGZJLTITXXDUWJA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:17.959833941 (69h41m17.830916996s) [noise=true , address=77.90.31.21:4877 , piece_id=QCETBSEYGNHWE27XPPSZQBANF7KDBFZIVLAZDRMMQSSLDO3GLGCA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 8 13:10:18.258818728 (69h41m18.129901763s) [noise=true , address=77.90.31.15:4699 , piece_id=TZQWWWCUSBWAKYNH75H6BBKM4JWUQIJUAUT3WDUY22AN6VIHMSXQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:10:21.319649531 (69h41m21.190732576s) [noise=true , address=145.79.227.30:4762 , piece_id=DBCI2WUILFAE5XRM672WKRXOSDLU2MXFYIGNVSGFJK7LOM4ZYDSQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:42.550056528 (69h41m42.421139563s) [noise=true , address=77.90.31.13:4627 , piece_id=S2Q7LJDREU3KRSBQTB3V2MTSZFHGAOTUODQBQ2KK34FHJNB3ODOA, seg_pos={0 0}, upload=61036]: Done uploading piece. err: <nil> Dec 8 13:10:42.550081766 (69h41m42.421164811s) [seg_pos={0 0}, upload=61036]: Segment reached optimal threshold of 13 pieces. Dec 8 13:10:42.550320301 (69h41m42.421403346s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Dec 8 13:10:42.550327524 (69h41m42.421410559s) [noise=true , address=77.90.31.21:4879 , piece_id=LRWDFB22W4U7BK6V6CMOQPQFUBD4LJVTHNXSGZ4MT5JHKAHXLRVQ, seg_pos={0 0}, upload=61036]: Long tail cancellation due to optimal threshold Dec 8 13:10:42.550349326 (69h41m42.421432361s) [seg_pos={0 0}, upload=61036]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:10:42.550428026 (69h41m42.421511091s) [upload=61036]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:10:42.550441801 (69h41m42.421524836s) [upload=61036]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:10:42.989209071 (69h41m42.860292116s) [upload=61036]: Done uploading segments... Dec 8 13:10:42.989231784 (69h41m42.860314829s) [upload=61036]: Upload finished. err: <nil> Duration: 30.002851092s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X...................................X...X.X.....] duration: 19.084106041s trace: https://jaeger.dev.storj.io/trace/3e0665108ee5b4b FAIL (2025-12-08 13:10:42 UTC) orbiter warning: context deadline exceeded Duration: 19.084138132s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.10966324s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/687b56ed9d6ebdc3 FAIL (2025-12-08 13:10:35 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 1.109679922s core satellite us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 503.869978ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:10:35 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 503.897822ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..XX.....X..X...........X......X..X......X....XX.XX.XXXXXXXX] duration: 5.001306194s trace: https://jaeger.dev.storj.io/trace/2a9d97488ac44af1 FAIL (2025-12-08 13:10:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0013348s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 469.628878ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:09:38 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 469.663814ms core satellite eu1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 712.407938ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/687b56ed9d6ebdc3 FAIL (2025-12-08 13:09:37 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 712.448715ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X...................................X...X.X....] duration: 26.886071918s trace: https://jaeger.dev.storj.io/trace/3e0665108ee5b4b FAIL (2025-12-08 13:09:36 UTC) orbiter warning: context deadline exceeded Duration: 26.886102797s edge gateway us-select-1

1
...
11719
11720
11721
11722
11723
...
17273