Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 4.837195753s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 03:12:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.837223948s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.....X..X....................................] duration: 5.6844938s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:12:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.684522064s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 259.017447ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/b66aeadd70f1f54 FAIL (2025-12-09 03:12:08 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout Duration: 259.048026ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.711409974s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:11:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.711438669s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 261.892604ms target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7fd16448e790267f FAIL (2025-12-09 03:11:48 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 261.92293ms edge linksharing bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.742453365s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:11:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.742483092s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.........X...X...................X...X.........XX.X.........] duration: 3.344214937s trace: https://jaeger.dev.storj.io/trace/57d90cc0034d6b88 FAIL (2025-12-09 03:11:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.344235726s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 1.160644759s trace: https://jaeger.dev.storj.io/trace/407d3ea9756acb8a FAIL (2025-12-09 03:11:35 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 1.160684754s edge authservice http bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [.................................................XX.........] duration: 241.421632ms target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7ca1ea61a644f062 FAIL (2025-12-09 03:11:30 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 241.453362ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 4.949564255s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 03:11:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.949593742s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.............X...................X...X...........XX.....X...] duration: 5.65899729s trace: https://jaeger.dev.storj.io/trace/58d1fad84827eb3 FAIL (2025-12-09 03:11:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.659023991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.....X..X...................................] duration: 5.699217368s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:11:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.699250019s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 1.161846545s trace: https://jaeger.dev.storj.io/trace/407d3ea9756acb8a FAIL (2025-12-09 03:11:12 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 1.161878866s edge authservice http bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..........X...X...................X...X.........XX.X........] duration: 3.63708165s trace: https://jaeger.dev.storj.io/trace/57d90cc0034d6b88 FAIL (2025-12-09 03:11:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.637114264s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 268.332907ms target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7fd16448e790267f FAIL (2025-12-09 03:11:08 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 268.364456ms edge linksharing bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 264.04005ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/b66aeadd70f1f54 FAIL (2025-12-09 03:11:03 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout Duration: 264.080337ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..............X...................X...X...........XX.....X..] duration: 5.750078286s trace: https://jaeger.dev.storj.io/trace/58d1fad84827eb3 FAIL (2025-12-09 03:11:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.75011159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.109883885s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:10:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.109967886s core uplink us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [..................................................XX........] duration: 247.981243ms target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7ca1ea61a644f062 FAIL (2025-12-09 03:10:52 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 248.008184ms edge linksharing bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...........X...X...................X...X.........XX.X.......] duration: 3.464232622s trace: https://jaeger.dev.storj.io/trace/57d90cc0034d6b88 FAIL (2025-12-09 03:10:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.464252018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.824584717s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:10:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.824601961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.352065975s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 03:10:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.352111932s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.....X..X..................................] duration: 5.750338791s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:10:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.750369308s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 273.857857ms target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7fd16448e790267f FAIL (2025-12-09 03:10:25 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 273.879077ms edge linksharing bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.763816816s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:10:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.763847975s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.825278612s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:10:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.825306786s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 1.18917224s trace: https://jaeger.dev.storj.io/trace/407d3ea9756acb8a FAIL (2025-12-09 03:10:10 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 1.189202075s edge authservice http bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.992283069s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:10:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.992341771s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...............X...................X...X...........XX.....X.] duration: 6.676083954s trace: https://jaeger.dev.storj.io/trace/58d1fad84827eb3 FAIL (2025-12-09 03:10:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.676101377s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [...................................................XX.......] duration: 244.900748ms target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7ca1ea61a644f062 FAIL (2025-12-09 03:09:55 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 244.926135ms edge linksharing bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [................X...................X...X...........XX.....X] duration: 1m0.000311812s trace: https://jaeger.dev.storj.io/trace/58d1fad84827eb3 FAIL (2025-12-09 03:09:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000424286s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 273.747939ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/b66aeadd70f1f54 FAIL (2025-12-09 03:09:51 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout Duration: 273.779589ms edge gateway vhost bgp us1
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 1.193026698s trace: https://jaeger.dev.storj.io/trace/407d3ea9756acb8a FAIL (2025-12-09 03:09:46 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 1.19305386s edge authservice http bgp us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 4.793771027s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 03:09:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.793800263s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.936825749s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:09:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.936858532s core uplink us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 270.25613ms target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7fd16448e790267f FAIL (2025-12-09 03:09:30 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 270.281667ms edge linksharing bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [............X...X...................X...X.........XX.X......] duration: 3.461636279s trace: https://jaeger.dev.storj.io/trace/57d90cc0034d6b88 FAIL (2025-12-09 03:09:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.461666015s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.....X..X.................................] duration: 5.705863804s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:09:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.705912828s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 1.162601543s trace: https://jaeger.dev.storj.io/trace/407d3ea9756acb8a FAIL (2025-12-09 03:09:22 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 1.162636149s edge authservice http bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.391084433s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:09:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.391111104s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [....................................................XX......] duration: 257.307432ms target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7ca1ea61a644f062 FAIL (2025-12-09 03:09:18 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 257.331568ms edge linksharing bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.............X...X...................X...X.........XX.X.....] duration: 3.507274364s trace: https://jaeger.dev.storj.io/trace/57d90cc0034d6b88 FAIL (2025-12-09 03:09:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.507390785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.848879082s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:09:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.848909741s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 1.172029616s trace: https://jaeger.dev.storj.io/trace/407d3ea9756acb8a FAIL (2025-12-09 03:08:52 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 1.172071736s edge authservice http bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 267.787749ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/b66aeadd70f1f54 FAIL (2025-12-09 03:08:39 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout Duration: 267.837863ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.................X...................X...X...........XX.....] duration: 5.762505201s trace: https://jaeger.dev.storj.io/trace/3af4de55303a6586 FAIL (2025-12-09 03:08:38 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 5.76253616s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 265.728878ms target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7fd16448e790267f FAIL (2025-12-09 03:08:36 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 265.777811ms edge linksharing bgp us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 4.90450135s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 03:08:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.904519596s core uplink eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [.....................................................XX.....] duration: 243.510932ms target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7ca1ea61a644f062 FAIL (2025-12-09 03:08:34 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 243.535097ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.082335668s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:08:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.082366578s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)