Orbiter History

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [.........................................................XX.] duration: 1.257667476s target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7ca1ea61a644f062 FAIL (2025-12-09 03:05:15 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 1.257682354s edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [..........................................................XX] duration: 30.001002403s target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7ca1ea61a644f062 FAIL (2025-12-09 03:05:14 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 30.00102687s edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 30.000745903s target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/7fd16448e790267f FAIL (2025-12-09 03:05:06 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 30.000772504s edge linksharing bgp us1
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.001050363s trace: https://jaeger.dev.storj.io/trace/407d3ea9756acb8a FAIL (2025-12-09 03:05:01 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 30.001065361s edge authservice http bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.950724269s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:04:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.950755378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.08420713s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 03:04:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.084470342s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.102669179s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:04:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.102688396s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...................X...X...................X...X.........XX.] duration: 3.482076777s trace: https://jaeger.dev.storj.io/trace/3f2c91e9527631de FAIL (2025-12-09 03:04:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.482113978s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 276.942376ms target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/b66aeadd70f1f54 FAIL (2025-12-09 03:04:40 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout Duration: 276.967695ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.................X.....X..X...........................] duration: 5.922079505s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:04:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.922097769s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......................X...................X...X...........XX] duration: 30.000720676s trace: https://jaeger.dev.storj.io/trace/3af4de55303a6586 FAIL (2025-12-09 03:04:34 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 30.000752718s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.................X.....X..X..........................] duration: 20.317273196s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:04:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 20.31731245s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000813444s trace: https://jaeger.dev.storj.io/trace/ba1eb8b10dd8550 FAIL (2025-12-09 03:04:31 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 30.000841307s edge authservice http bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 30.001681529s target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/65273687acb98a9f FAIL (2025-12-09 03:04:20 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout Duration: 30.001705365s edge linksharing bgp eu1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001498567s target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/b66aeadd70f1f54 FAIL (2025-12-09 03:04:14 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout Duration: 30.001533113s edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.152646291s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 03:04:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.152674424s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.852592537s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:04:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.852613447s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......................X...................X...X...........X] duration: 30.000743411s trace: https://jaeger.dev.storj.io/trace/740d6f40e6afa4e3 FAIL (2025-12-09 03:04:04 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout Duration: 30.000772406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m27.582529951s trace: https://jaeger.dev.storj.io/trace/5f4ccc758f414b33 FAIL (2025-12-09 03:04:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1m27.582556673s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.438491583s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:03:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.438514607s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....................X...X...................X...X.........XX] duration: 5.001169822s trace: https://jaeger.dev.storj.io/trace/3f2c91e9527631de FAIL (2025-12-09 03:03:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001191885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.................X.....X..X.........................] duration: 5.665859716s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:03:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.665896247s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.597847099s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 03:03:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.597913758s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.774140444s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:02:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.774172534s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....................X...X...................X...X.........X] duration: 5.001312045s trace: https://jaeger.dev.storj.io/trace/379948747b10cc30 FAIL (2025-12-09 03:02:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001342452s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.................X.....X..X........................] duration: 5.71252628s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:02:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.712556056s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......................X...X...................X...X.........] duration: 3.449806498s trace: https://jaeger.dev.storj.io/trace/3cdd0b10f9b1f962 FAIL (2025-12-09 03:02:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.449832278s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................X...................X...X...........] duration: 5.63156828s trace: https://jaeger.dev.storj.io/trace/73c4211bd9c002a7 FAIL (2025-12-09 03:02:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.631597336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.................X.....X..X.......................] duration: 5.633397896s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:02:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.633426241s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......................X...X...................X...X........] duration: 3.522533384s trace: https://jaeger.dev.storj.io/trace/3cdd0b10f9b1f962 FAIL (2025-12-09 03:02:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.52255249s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.121122505s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 03:02:12 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.121153534s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................X...X...................X...X.......] duration: 4.354627113s trace: https://jaeger.dev.storj.io/trace/3cdd0b10f9b1f962 FAIL (2025-12-09 03:01:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.354729468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.806595804s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:01:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.806624629s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................X...................X...X..........] duration: 5.651710784s trace: https://jaeger.dev.storj.io/trace/73c4211bd9c002a7 FAIL (2025-12-09 03:01:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.651747165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.................X.....X..X......................] duration: 5.71671799s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:01:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.716742828s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 4.872298441s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 03:01:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.872318869s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........................X...................X...X.........] duration: 5.918827293s trace: https://jaeger.dev.storj.io/trace/73c4211bd9c002a7 FAIL (2025-12-09 03:01:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.918862219s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........................X...X...................X...X......] duration: 3.434332324s trace: https://jaeger.dev.storj.io/trace/3cdd0b10f9b1f962 FAIL (2025-12-09 03:00:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.434363544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.................X.....X..X.....................] duration: 5.738497092s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:00:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.738524194s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.169422436s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 03:00:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.169462541s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.973938473s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 03:00:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.973972439s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........................X...X...................X...X.....] duration: 3.464612933s trace: https://jaeger.dev.storj.io/trace/3cdd0b10f9b1f962 FAIL (2025-12-09 03:00:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.464641749s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........................X...................X...X........] duration: 5.861095912s trace: https://jaeger.dev.storj.io/trace/73c4211bd9c002a7 FAIL (2025-12-09 03:00:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.861127853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.................X.....X..X....................] duration: 6.016534704s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 03:00:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.01659082s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........................X...X...................X...X....] duration: 3.490470029s trace: https://jaeger.dev.storj.io/trace/3cdd0b10f9b1f962 FAIL (2025-12-09 03:00:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.49050173s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............................X...................X...X.......] duration: 5.900889797s trace: https://jaeger.dev.storj.io/trace/73c4211bd9c002a7 FAIL (2025-12-09 02:59:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.900916287s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.793576801s trace: https://jaeger.dev.storj.io/trace/5b7efc901355db5e FAIL (2025-12-09 02:59:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581295090 (83h6m52.452378125s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.581634846 (83h6m52.452717891s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.582888624 (83h6m52.453971659s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583267836 (83h6m52.454350871s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583663720 (83h6m52.454746766s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.583984842 (83h6m52.455067887s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584326653 (83h6m52.455409698s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.584654216 (83h6m52.455737251s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585020815 (83h6m52.45610386s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.585435474 (83h6m52.456518519s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586429306 (83h6m52.457512351s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.586955358 (83h6m52.458038403s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588208955 (83h6m52.459292s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.588553080 (83h6m52.459636115s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:52.589933539 (83h6m52.461016574s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Uploading piece... Dec 9 02:35:54.045728988 (83h6m53.916812023s) [upload=72598]: Next returned nil segment Dec 9 02:35:54.045753754 (83h6m53.916836799s) [upload=72598]: Waiting for error group managing segments... Dec 9 02:35:54.429970583 (83h6m54.301053618s) [noise=true , address=145.79.227.8:4334 , piece_id=DMWK5WVDG7VZWAWBV6UILCHWA57J2C2D7DHMXTJBM4UWJTF3OWPA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.444386584 (83h6m54.315469629s) [noise=true , address=145.79.227.5:4233 , piece_id=D4V4LNQ7QOP3OQDTWDBUXHGHAS7IYPXARV3AYCPKGWZTEO6OTICA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.469193360 (83h6m54.340276395s) [noise=true , address=145.79.227.38:4996 , piece_id=JUFG4APIQSTWSS4MQ6JUBKLTEOHVV64YU6ZTPLMWMRF346HBO6GQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.472137816 (83h6m54.343220862s) [noise=true , address=145.79.227.22:4540 , piece_id=2ATNCIUNDNDD57MSDWU356YSJY4UMRZSQFLJGXZQQWQSVO7E7OJA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.484648119 (83h6m54.355731204s) [noise=true , address=145.79.227.51:4436 , piece_id=NKWQM7OVKVCQCEFF6WSY3E653UMT5RBOWDX6WSQAD7NJ6VK5RKMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.489783557 (83h6m54.360866602s) [noise=true , address=145.79.227.26:4656 , piece_id=IC4AUOX2QE44UDPFEWAOFDY4W25RX24DKU4WPFKSQWM74EJNM75A, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.596419028 (83h6m54.467502073s) [noise=true , address=77.90.31.21:4887 , piece_id=IXSCAD7EUGK3ZZ4H6R3F5N2WF7MOI4DBTAVVFFIBUT2NKRJCBULA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.598540137 (83h6m54.469623172s) [noise=true , address=77.90.31.15:4714 , piece_id=LP7HCHJ3TXO5OJWGDJFV5RMKASVDMLBB5G6NKTZKLOBHHVMG5T2Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.602190628 (83h6m54.473273674s) [noise=true , address=45.91.116.8:4046 , piece_id=SCSARQHG5I6JB4365O5MPJRCJAA6UKHLR6XHY6EQYOH5CRMR4ZQQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.604555752 (83h6m54.475638797s) [noise=true , address=45.91.116.13:4164 , piece_id=XV3OXMXPJTFXBFMND7EMW3NUPW4P2ZQVL63CTXPDCMMZDTNOZMMQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.623163304 (83h6m54.494246349s) [noise=true , address=77.90.31.19:4875 , piece_id=27ACR5MATWJSBAQBMN6MSFA4LJODOW2FPG4WDLBJASIRVBOGI73Q, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.630125902 (83h6m54.501208947s) [noise=true , address=77.90.31.14:4691 , piece_id=CHKKJVWU5IFXLODN2RDB3SILIWGBZGJ7RLPNEDWYFIFIUGZIDKPQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.644849627 (83h6m54.515932672s) [noise=true , address=45.91.116.10:4104 , piece_id=TGQVBFKOVCHHS76B6HQBX6RNCNHV7XBE3JW2CUCTEO54VV7OVLWQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661873844 (83h6m54.532956889s) [noise=true , address=45.91.116.14:4214 , piece_id=U5UMWUHBQELI7M23VDC2GJNGYG676B4LT7Z7Z5DHGOYRSW3JOSHQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: <nil> Dec 9 02:35:54.661936824 (83h6m54.533019869s) [seg_pos={0 0}, upload=72598]: Segment reached optimal threshold of 14 pieces. Dec 9 02:35:54.662405798 (83h6m54.533488843s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Dfm6iHQbMhj1YWLu3UboTCtZ2hq87aiYUgH7Kig3pDUrPeFj4): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662415215 (83h6m54.53349826s) [noise=true , address=45.91.116.9:4272 , piece_id=4DUWR3IDCLYLEGOK2DC7GILHO3UW24H34PG36PVB6UGYDOPUHCVA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662471652 (83h6m54.533554697s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35816->64.49.13.169:4706: use of closed network connection Dec 9 02:35:54.662517330 (83h6m54.533600375s) [noise=true , address=64.49.13.169:4706 , piece_id=IPMPTX7W572TPX34J5I6VGLSQMB44MYQISE6WYTUN4G2BJXMDNNA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662678426 (83h6m54.533761461s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: write tcp 188.166.225.170:42636->77.90.31.16:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:35:54.662687404 (83h6m54.533770449s) [noise=true , address=77.90.31.16:4766 , piece_id=5Z4SH4GYB3QB2CBHF3A2NMUSH5QI575GKN7B7EFKL6NBF7X7M3ZQ, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662891562 (83h6m54.533974597s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Dec 9 02:35:54.662901071 (83h6m54.533984106s) [noise=true , address=77.90.31.13:4640 , piece_id=C3QWIVLOZSJDB2MWMEBEPPLVPF4H3HHBCJH7ZWAF64BWDQOI65GA, seg_pos={0 0}, upload=72598]: Long tail cancellation due to optimal threshold Dec 9 02:35:54.662923623 (83h6m54.534006658s) [seg_pos={0 0}, upload=72598]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 02:35:54.663067688 (83h6m54.534150723s) [upload=72598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:35:54.663073940 (83h6m54.534156985s) [upload=72598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:38:38.394271693 (83h9m38.265354758s) [upload=72598]: Done uploading segments... Dec 9 02:38:38.394386772 (83h9m38.265469807s) [upload=72598]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.793604614s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 4.927564135s trace: https://jaeger.dev.storj.io/trace/78baf37c16400f4c FAIL (2025-12-09 02:59:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.927597338s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............................X...X...................X...X...] duration: 3.499073211s trace: https://jaeger.dev.storj.io/trace/3cdd0b10f9b1f962 FAIL (2025-12-09 02:59:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.499097337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.................X.....X..X...................] duration: 5.694741172s trace: https://jaeger.dev.storj.io/trace/47af5af0eb589e37 FAIL (2025-12-09 02:59:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments... Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment... Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment. Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece... Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments... Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil> Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000 Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments... Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.694773073s core uplink us-select-1

Orbiter Check History

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)