Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X...................X...X.........XX.X.............X.......] duration: 3.475320529s trace: https://jaeger.dev.storj.io/trace/5a015f2219b688c3 FAIL (2025-12-09 03:19:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.475333663s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....X...................X...X...........XX.....X...X........] duration: 5.980482699s trace: https://jaeger.dev.storj.io/trace/5da432e0ce6eb8e FAIL (2025-12-09 03:19:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.980501444s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.774937712s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:19:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.774963871s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X...................X...X.........XX.X.............X......] duration: 3.398421495s trace: https://jaeger.dev.storj.io/trace/5a015f2219b688c3 FAIL (2025-12-09 03:19:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.398453366s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 1.223259878s trace: https://jaeger.dev.storj.io/trace/407d3ea9756acb8a FAIL (2025-12-09 03:19:07 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 1.223307349s edge authservice http bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.....X..X...............................................] duration: 5.758765378s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:19:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.758810524s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.996969804s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:18:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.997001454s core uplink us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 269.873951ms target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7fd16448e790267f FAIL (2025-12-09 03:18:56 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 269.903586ms edge linksharing bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [......................................XX....................] duration: 248.742171ms target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7ca1ea61a644f062 FAIL (2025-12-09 03:18:51 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 248.776368ms edge linksharing bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X...................X...X.........XX.X.............X.....] duration: 3.418202243s trace: https://jaeger.dev.storj.io/trace/5a015f2219b688c3 FAIL (2025-12-09 03:18:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.418230246s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 268.351341ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/b66aeadd70f1f54 FAIL (2025-12-09 03:18:41 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout Duration: 268.378664ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.842600986s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:18:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.842630983s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 4.874345615s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 03:18:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.874376233s core uplink eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 268.694796ms target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7fd16448e790267f FAIL (2025-12-09 03:18:31 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 268.713981ms edge linksharing bgp us1
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 1.183415146s trace: https://jaeger.dev.storj.io/trace/407d3ea9756acb8a FAIL (2025-12-09 03:18:31 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 1.183444552s edge authservice http bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.723680541s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:18:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.723769961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.664028189s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:18:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.66406082s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....X...................X...X...........XX.....X...X.......] duration: 5.862857718s trace: https://jaeger.dev.storj.io/trace/5da432e0ce6eb8e FAIL (2025-12-09 03:18:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.862878477s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.....X..X..............................................] duration: 5.717541859s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:18:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.717572949s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [.......................................XX...................] duration: 255.09555ms target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7ca1ea61a644f062 FAIL (2025-12-09 03:18:15 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 255.118133ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.816078784s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:18:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.816104815s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [......X...................X...X...........XX.....X...X......] duration: 5.982301203s trace: https://jaeger.dev.storj.io/trace/5da432e0ce6eb8e FAIL (2025-12-09 03:18:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.982332353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 4.872236039s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 03:18:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.872266878s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X...X...................X...X.........XX.X.............X....] duration: 3.368219729s trace: https://jaeger.dev.storj.io/trace/5a015f2219b688c3 FAIL (2025-12-09 03:17:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.368275565s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 272.002033ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/b66aeadd70f1f54 FAIL (2025-12-09 03:17:52 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout Duration: 272.033412ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.671764914s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:17:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.671791385s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X...X...................X...X.........XX.X.............X...] duration: 3.537662016s trace: https://jaeger.dev.storj.io/trace/5a015f2219b688c3 FAIL (2025-12-09 03:17:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.537733532s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [........................................XX..................] duration: 250.518642ms target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7ca1ea61a644f062 FAIL (2025-12-09 03:17:30 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 250.544731ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 273.356213ms target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7fd16448e790267f FAIL (2025-12-09 03:17:27 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 273.399265ms edge linksharing bgp us1
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 1.18232911s trace: https://jaeger.dev.storj.io/trace/407d3ea9756acb8a FAIL (2025-12-09 03:17:27 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 1.182356852s edge authservice http bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.680033254s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:17:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.680070887s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.910768138s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:17:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.910797594s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 272.510672ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/b66aeadd70f1f54 FAIL (2025-12-09 03:17:13 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout Duration: 272.539969ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.......X...................X...X...........XX.....X...X.....] duration: 5.699262502s trace: https://jaeger.dev.storj.io/trace/5da432e0ce6eb8e FAIL (2025-12-09 03:17:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.699294243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.785148046s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:17:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.785179887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.....X..X.............................................] duration: 5.721760084s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:17:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.721781415s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.127334374s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 03:17:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.127367326s core uplink eu1
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 1.170938607s trace: https://jaeger.dev.storj.io/trace/407d3ea9756acb8a FAIL (2025-12-09 03:16:56 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 1.170957443s edge authservice http bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 263.398992ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/b66aeadd70f1f54 FAIL (2025-12-09 03:16:52 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout Duration: 263.421825ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.....X..X............................................] duration: 15.8026275s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:16:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 15.802663598s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.568118699s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:16:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.568148175s core uplink us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 263.765118ms target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7fd16448e790267f FAIL (2025-12-09 03:16:33 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 263.792781ms edge linksharing bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..X...X...................X...X.........XX.X.............X..] duration: 3.46257576s trace: https://jaeger.dev.storj.io/trace/5a015f2219b688c3 FAIL (2025-12-09 03:16:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462636195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.....X..X...........................................] duration: 5.700171093s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:16:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.700221948s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 272.684824ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/b66aeadd70f1f54 FAIL (2025-12-09 03:16:21 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout Duration: 272.713298ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [.........................................XX.................] duration: 255.645586ms target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7ca1ea61a644f062 FAIL (2025-12-09 03:16:21 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 255.674312ms edge linksharing bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [........X...................X...X...........XX.....X...X....] duration: 5.660859386s trace: https://jaeger.dev.storj.io/trace/5da432e0ce6eb8e FAIL (2025-12-09 03:16:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.660887338s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 1.157446215s trace: https://jaeger.dev.storj.io/trace/407d3ea9756acb8a FAIL (2025-12-09 03:16:16 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 1.157474922s edge authservice http bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [..........................................XX................] duration: 263.09913ms target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7ca1ea61a644f062 FAIL (2025-12-09 03:16:05 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 263.131892ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.849199675s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:15:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.849231645s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)