Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........X...X.X......X.XX...X.X.X....X............X....X.X] duration: 30.00342362s trace: https://jaeger.dev.storj.io/trace/4f6e48f588fcce83 FAIL (2025-11-21 11:42:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 2N4UQ5UNTKSQQMFZEG2UFSNE5AISQ7I4O5JJYX7TPZN32BUG4TDA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: FRCQYZVW5GX7YURDKYLDBLOSRJPY7MYOVL5W7LVJ3QV75L53FAEQ) context deadline exceeded ======LOGS====== 1 11:42:27.582346605 (291h38m26.133817066s) [upload=256832]: Starting upload Nov 21 11:42:27.582419664 (291h38m26.133890125s) [upload=256832]: Done starting upload Nov 21 11:42:27.582484858 (291h38m26.133955319s) [upload=256832]: Uploading segments... Nov 21 11:42:27.582628000 (291h38m26.134098451s) [upload=256832]: Got next segment. Inline: false Nov 21 11:42:27.582668327 (291h38m26.134138788s) [upload=256832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:42:27.582672956 (291h38m26.134143407s) [upload=256832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:42:27.827064344 (291h38m26.378534806s) [seg_pos={0 0}, upload=256832]: Begin upload segment... Nov 21 11:42:27.827119079 (291h38m26.37858954s) [seg_pos={0 0}, upload=256832]: Done begin upload segment. Nov 21 11:42:27.827153535 (291h38m26.378623996s) [noise=true , address=77.90.31.17:4770 , piece_id=RWJFMY524OVK57SNMQ7YFMPQRID6QS2SHUX57KE6PBIRDTVRI6XQ, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.827191507 (291h38m26.378661958s) [noise=true , address=145.79.227.22:4509 , piece_id=XFR3JCYUNAA6XNRALD2FKC3MAWUCP7SZTLO64YFQO3DTLNFBFQFA, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.827949740 (291h38m26.379420201s) [noise=true , address=45.91.116.8:4048 , piece_id=2N4UQ5UNTKSQQMFZEG2UFSNE5AISQ7I4O5JJYX7TPZN32BUG4TDA, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.828041796 (291h38m26.379512257s) [noise=true , address=145.79.227.30:4762 , piece_id=X4U5XOY3WCTLSFVMPO46IYXWLS7CXBKXVYJTR5M3EYUSN4H4FMIQ, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.828655685 (291h38m26.380126136s) [noise=true , address=45.91.116.4:4940 , piece_id=DTQHJJGXOXE6B2VHK5JEMAVKOSWNDRTG3CDGSBKMQI3AOV3UWPMQ, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.828722051 (291h38m26.380192512s) [noise=true , address=145.79.227.20:4439 , piece_id=D4TGIE7NGGWLO3Z4PUTS45BETLZYMWKWGZUZWV27N6YSPZVUUM7A, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.829134015 (291h38m26.380604476s) [noise=true , address=145.79.227.20:4437 , piece_id=6NLNBHESW2YOFY6NPNLIY4OJYSGM4TYO6DD6Z3A5P2CSFV4EH2RQ, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.829260827 (291h38m26.380731288s) [noise=true , address=145.79.227.25:4617 , piece_id=BGWXWPS6PRMX7QY4526E6ETXZ2ADNFJ6R3N67YXYUPMZU72E2BTA, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.829623769 (291h38m26.38109423s) [noise=true , address=77.90.31.15:4697 , piece_id=ZXQUVQZSN27ML5WQ2GOO3BB5G2U75CLEVNSSBG6MZRNH4D6IRVXA, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.830286991 (291h38m26.381757483s) [noise=true , address=45.91.116.14:4190 , piece_id=FRCQYZVW5GX7YURDKYLDBLOSRJPY7MYOVL5W7LVJ3QV75L53FAEQ, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.830564760 (291h38m26.382035221s) [noise=true , address=145.79.227.31:4798 , piece_id=DDWZBNIJSDLNGZXNUSO644NYGG5SNMV7NE4K6L56DPPT3XTAWOZQ, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.830886302 (291h38m26.382356763s) [noise=true , address=145.79.227.14:4839 , piece_id=FG26V3C4G3RKOTNPI4HBWY3HD6YQNGF2W5PJVKC5OQPTBDYGZYYQ, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.831939898 (291h38m26.383410359s) [noise=true , address=145.79.227.11:4730 , piece_id=NF243ZYVKX557RFPWNTY6LY3AR3VCQJ3BMFQIUOIV6YRAGEJJTAA, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.832166560 (291h38m26.383637021s) [noise=true , address=77.90.31.11:4554 , piece_id=VUDSHBH7QY5IQ57UP4SRKYZ5QCHRRYWU3E4ZDAZTF65LU3ME2WOA, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.833257047 (291h38m26.384727508s) [noise=true , address=145.79.227.27:4689 , piece_id=6INJ622RXLRR3JMMWVFEJKEOCQKNUBM4CCHID7V7FIRDPJ2EHUPA, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:27.834108598 (291h38m26.385579059s) [noise=true , address=145.79.227.28:4726 , piece_id=VNCH5X2XDC7U7I2EICAU66EPJ74B4LZGXQDAFKE4HREPMCL7XHTA, seg_pos={0 0}, upload=256832]: Uploading piece... Nov 21 11:42:29.443724882 (291h38m27.995195343s) [upload=256832]: Next returned nil segment Nov 21 11:42:29.443741404 (291h38m27.995211875s) [upload=256832]: Waiting for error group managing segments... Nov 21 11:42:29.825233406 (291h38m28.376703857s) [noise=true , address=145.79.227.30:4762 , piece_id=X4U5XOY3WCTLSFVMPO46IYXWLS7CXBKXVYJTR5M3EYUSN4H4FMIQ, seg_pos={0 0}, upload=256832]: Done uploading piece. err: <nil> Nov 21 11:42:29.828134501 (291h38m28.379604962s) [noise=true , address=145.79.227.31:4798 , piece_id=DDWZBNIJSDLNGZXNUSO644NYGG5SNMV7NE4K6L56DPPT3XTAWOZQ, seg_pos={0 0}, upload=256832]: Done uploading piece. err: <nil> Nov 21 11:42:29.828916299 (291h38m28.38038676s) [noise=true , address=145.79.227.25:4617 , piece_id=BGWXWPS6PRMX7QY4526E6ETXZ2ADNFJ6R3N67YXYUPMZU72E2BTA, seg_pos={0 0}, upload=256832]: Done uploading piece. err: <nil> Nov 21 11:42:29.830188623 (291h38m28.381659084s) [noise=true , address=145.79.227.20:4439 , piece_id=D4TGIE7NGGWLO3Z4PUTS45BETLZYMWKWGZUZWV27N6YSPZVUUM7A, seg_pos={0 0}, upload=256832]: Done uploading piece. err: <nil> Nov 21 11:42:29.831138010 (291h38m28.382608471s) [noise=true , address=145.79.227.11:4730 , piece_id=NF243ZYVKX557RFPWNTY6LY3AR3VCQJ3BMFQIUOIV6YRAGEJJTAA, seg_pos={0 0}, upload=256832]: Done uploading piece. err: <nil> Nov 21 11:42:29.831608286 (291h38m28.383078737s) [noise=true , address=145.79.227.20:4437 , piece_id=6NLNBHESW2YOFY6NPNLIY4OJYSGM4TYO6DD6Z3A5P2CSFV4EH2RQ, seg_pos={0 0}, upload=256832]: Done uploading piece. err: <nil> Nov 21 11:42:29.839411715 (291h38m28.390882166s) [noise=true , address=145.79.227.14:4839 , piece_id=FG26V3C4G3RKOTNPI4HBWY3HD6YQNGF2W5PJVKC5OQPTBDYGZYYQ, seg_pos={0 0}, upload=256832]: Done uploading piece. err: <nil> Nov 21 11:42:29.839834861 (291h38m28.391305332s) [noise=true , address=145.79.227.22:4509 , piece_id=XFR3JCYUNAA6XNRALD2FKC3MAWUCP7SZTLO64YFQO3DTLNFBFQFA, seg_pos={0 0}, upload=256832]: Done uploading piece. err: <nil> Nov 21 11:42:29.846124898 (291h38m28.397595359s) [noise=true , address=145.79.227.27:4689 , piece_id=6INJ622RXLRR3JMMWVFEJKEOCQKNUBM4CCHID7V7FIRDPJ2EHUPA, seg_pos={0 0}, upload=256832]: Done uploading piece. err: <nil> Nov 21 11:42:29.854921609 (291h38m28.40639206s) [noise=true , address=145.79.227.28:4726 , piece_id=VNCH5X2XDC7U7I2EICAU66EPJ74B4LZGXQDAFKE4HREPMCL7XHTA, seg_pos={0 0}, upload=256832]: Done uploading piece. err: <nil> Nov 21 11:42:30.404759674 (291h38m28.956230146s) [noise=true , address=45.91.116.8:4048 , piece_id=2N4UQ5UNTKSQQMFZEG2UFSNE5AISQ7I4O5JJYX7TPZN32BUG4TDA, seg_pos={0 0}, upload=256832]: Done uploading piece. err: <nil> Nov 21 11:42:30.531857242 (291h38m29.083327693s) [noise=true , address=45.91.116.14:4190 , piece_id=FRCQYZVW5GX7YURDKYLDBLOSRJPY7MYOVL5W7LVJ3QV75L53FAEQ, seg_pos={0 0}, upload=256832]: Done uploading piece. err: <nil> Nov 21 11:42:30.758748116 (291h38m29.310218577s) [noise=true , address=77.90.31.11:4554 , piece_id=VUDSHBH7QY5IQ57UP4SRKYZ5QCHRRYWU3E4ZDAZTF65LU3ME2WOA, seg_pos={0 0}, upload=256832]: Done uploading piece. err: <nil> Nov 21 11:42:30.758774556 (291h38m29.310245017s) [seg_pos={0 0}, upload=256832]: Segment reached optimal threshold of 13 pieces. Nov 21 11:42:30.759205336 (291h38m29.310675797s) [noise=true , address=77.90.31.17:4770 , piece_id=RWJFMY524OVK57SNMQ7YFMPQRID6QS2SHUX57KE6PBIRDTVRI6XQ, seg_pos={0 0}, upload=256832]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 11:42:30.759245923 (291h38m29.310716374s) [noise=true , address=77.90.31.17:4770 , piece_id=RWJFMY524OVK57SNMQ7YFMPQRID6QS2SHUX57KE6PBIRDTVRI6XQ, seg_pos={0 0}, upload=256832]: Long tail cancellation due to optimal threshold Nov 21 11:42:30.759290920 (291h38m29.310761381s) [noise=true , address=77.90.31.15:4697 , piece_id=ZXQUVQZSN27ML5WQ2GOO3BB5G2U75CLEVNSSBG6MZRNH4D6IRVXA, seg_pos={0 0}, upload=256832]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Nov 21 11:42:30.759301820 (291h38m29.310772281s) [noise=true , address=77.90.31.15:4697 , piece_id=ZXQUVQZSN27ML5WQ2GOO3BB5G2U75CLEVNSSBG6MZRNH4D6IRVXA, seg_pos={0 0}, upload=256832]: Long tail cancellation due to optimal threshold Nov 21 11:42:30.759587844 (291h38m29.311058306s) [noise=true , address=45.91.116.4:4940 , piece_id=DTQHJJGXOXE6B2VHK5JEMAVKOSWNDRTG3CDGSBKMQI3AOV3UWPMQ, seg_pos={0 0}, upload=256832]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:56388->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.4:4940: use of closed network connection Nov 21 11:42:30.759598234 (291h38m29.311068695s) [noise=true , address=45.91.116.4:4940 , piece_id=DTQHJJGXOXE6B2VHK5JEMAVKOSWNDRTG3CDGSBKMQI3AOV3UWPMQ, seg_pos={0 0}, upload=256832]: Long tail cancellation due to optimal threshold Nov 21 11:42:30.759624644 (291h38m29.311095106s) [seg_pos={0 0}, upload=256832]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:42:30.759725727 (291h38m29.311196188s) [upload=256832]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:42:30.759734524 (291h38m29.311204985s) [upload=256832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:42:31.046104166 (291h38m29.597574617s) [upload=256832]: Done uploading segments... Nov 21 11:42:31.046167967 (291h38m29.597638418s) [upload=256832]: Upload finished. err: <nil> Duration: 30.003455311s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 259.930998ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:42:47 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 259.960423ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 257.491764ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:42:41 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 257.515649ms edge linksharing bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX........] duration: 3.369842511s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:42:41 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.369868s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 257.042136ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:42:29 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 257.070812ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 242.350201ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:42:27 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 242.37647ms edge gateway vhost bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 258.090173ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:42:24 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 258.121573ms edge linksharing bgp us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.X........................] duration: 5.927953887s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:42:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.927973805s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 255.927495ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:42:02 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 255.989943ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...........X...X.X......X.XX...X.X.X....X............X....X.] duration: 6.429929876s trace: https://jaeger.dev.storj.io/trace/69e1ebd1c45373c FAIL (2025-11-21 11:41:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 3BI22NJKKK46PRFZ7MQ5VRRUMWTMCX2BU5DUVUF2XMGNWUZKBOLQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: MAXQEDRM3PAP3VTYXCP4UPZKMJ7EGYW7KE3ZX2EKCZBDJGNQ24WQ) context deadline exceeded ======LOGS====== Nov 21 11:40:44.490365952 (291h36m43.041836423s) [upload=256808]: Starting upload Nov 21 11:40:44.490456043 (291h36m43.041926504s) [upload=256808]: Done starting upload Nov 21 11:40:44.490848269 (291h36m43.04231874s) [upload=256808]: Uploading segments... Nov 21 11:40:44.490910358 (291h36m43.042380829s) [upload=256808]: Got next segment. Inline: false Nov 21 11:40:44.490949793 (291h36m43.042420274s) [upload=256808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:40:44.490956717 (291h36m43.042427198s) [upload=256808]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:40:44.735648634 (291h36m43.287119095s) [seg_pos={0 0}, upload=256808]: Begin upload segment... Nov 21 11:40:44.735702587 (291h36m43.287173048s) [seg_pos={0 0}, upload=256808]: Done begin upload segment. Nov 21 11:40:44.735744437 (291h36m43.287214908s) [noise=true , address=77.90.31.16:4734 , piece_id=VUJDCDUQ3K5FIMDDN2Q3GI5VFXBUZJTPB5RDJYXIUY2YDYVZFWNA, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.735847072 (291h36m43.287317533s) [noise=true , address=145.79.227.8:4331 , piece_id=A26GAR4UEBCZ2DCSS3XUSV3AXDQDJ4W5L4UROJXK7GABH44BGBMA, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.736447876 (291h36m43.287918327s) [noise=true , address=145.79.227.15:4875 , piece_id=SDA2NA6L7A37XBPXCOJVZAOL4JHT3E4774XZM6VVNDZKZO3FCDHQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.736775801 (291h36m43.288246262s) [noise=true , address=77.90.31.10:4519 , piece_id=3BI22NJKKK46PRFZ7MQ5VRRUMWTMCX2BU5DUVUF2XMGNWUZKBOLQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.737089789 (291h36m43.28856025s) [noise=true , address=145.79.227.11:4730 , piece_id=FD5CGPBY3EPFNQJLF2IAZAUSZ4XSHDZVSRAUDNGOLUDFZRSTML4A, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.737427411 (291h36m43.288897892s) [noise=true , address=45.91.116.7:4009 , piece_id=44SOTGVCG4ZAUNMNNGFBPJGI464X6E4XXEOOZ3W2KJYPFYIM6UHQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.737922874 (291h36m43.289393325s) [noise=true , address=77.90.31.19:4841 , piece_id=H4DS5OXB4H636NMNKQ3O3BPBPAIMWKD3JLF4JZBYD4HCLZYZZEQQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.738266609 (291h36m43.28973706s) [noise=true , address=145.79.227.25:4618 , piece_id=QOKGVWIN2JF3E2YBGCEJR5ORVRMWUXA2K4Q6RMA4MWCJ7J5Z65PQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.738800386 (291h36m43.290270837s) [noise=true , address=45.91.116.10:4082 , piece_id=MRQS5HQWNHDD5QFM47OTFX5TDFZSU5JPHO7RT2Z2AFGEOALZZ44Q, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.739403194 (291h36m43.290873655s) [noise=true , address=145.79.227.28:4726 , piece_id=ULYH4ZQUNP7RBA7WEWFUDRP3A5BBOLNAZFKP7BS6JBDN44PSTX3A, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.739811351 (291h36m43.291281812s) [noise=true , address=145.79.227.30:4763 , piece_id=QDYPZHXOVU5BEGZJFOLZV7FKSGI37HNUVOCOHQA66DFGVSSDAIAQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.740155647 (291h36m43.291626098s) [noise=true , address=45.91.116.14:4190 , piece_id=MAXQEDRM3PAP3VTYXCP4UPZKMJ7EGYW7KE3ZX2EKCZBDJGNQ24WQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.740623147 (291h36m43.292093598s) [noise=true , address=145.79.227.11:4731 , piece_id=4EFQX5WQGOB32XDSDDVLJE6276KS6I43OAFK22H42V7ZNCIX7CNA, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.741905819 (291h36m43.29337628s) [noise=true , address=145.79.227.31:4799 , piece_id=CWVNBV2KKS4CSUGALZXS3T4NSOD4WSWBFEDRGSSK3WUWSZT3EPRQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.747098617 (291h36m43.298569078s) [noise=true , address=145.79.227.24:4581 , piece_id=OTKFUS6CNHE2HUSOPAFUB7E7MQ2ZKJE5YC3U27TFGKAVWJRM6YJQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.748709514 (291h36m43.300179975s) [noise=true , address=145.79.227.14:4839 , piece_id=NX6AERKM6L2RX3JN5LZQ4EBBDTL6BE2ZLL4B25JZJU2DYN2LNDOQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:46.443828492 (291h36m44.995298953s) [upload=256808]: Next returned nil segment Nov 21 11:40:46.443844232 (291h36m44.995314694s) [upload=256808]: Waiting for error group managing segments... Nov 21 11:40:46.724509292 (291h36m45.275979753s) [noise=true , address=145.79.227.11:4731 , piece_id=4EFQX5WQGOB32XDSDDVLJE6276KS6I43OAFK22H42V7ZNCIX7CNA, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.731925745 (291h36m45.283396206s) [noise=true , address=145.79.227.11:4730 , piece_id=FD5CGPBY3EPFNQJLF2IAZAUSZ4XSHDZVSRAUDNGOLUDFZRSTML4A, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.732400628 (291h36m45.283871089s) [noise=true , address=145.79.227.8:4331 , piece_id=A26GAR4UEBCZ2DCSS3XUSV3AXDQDJ4W5L4UROJXK7GABH44BGBMA, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.734032606 (291h36m45.285503067s) [noise=true , address=145.79.227.24:4581 , piece_id=OTKFUS6CNHE2HUSOPAFUB7E7MQ2ZKJE5YC3U27TFGKAVWJRM6YJQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.741638760 (291h36m45.293109221s) [noise=true , address=145.79.227.25:4618 , piece_id=QOKGVWIN2JF3E2YBGCEJR5ORVRMWUXA2K4Q6RMA4MWCJ7J5Z65PQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.746594867 (291h36m45.298065328s) [noise=true , address=145.79.227.28:4726 , piece_id=ULYH4ZQUNP7RBA7WEWFUDRP3A5BBOLNAZFKP7BS6JBDN44PSTX3A, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.752194069 (291h36m45.30366453s) [noise=true , address=145.79.227.15:4875 , piece_id=SDA2NA6L7A37XBPXCOJVZAOL4JHT3E4774XZM6VVNDZKZO3FCDHQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.753504845 (291h36m45.304975306s) [noise=true , address=145.79.227.14:4839 , piece_id=NX6AERKM6L2RX3JN5LZQ4EBBDTL6BE2ZLL4B25JZJU2DYN2LNDOQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.755097948 (291h36m45.306568399s) [noise=true , address=145.79.227.30:4763 , piece_id=QDYPZHXOVU5BEGZJFOLZV7FKSGI37HNUVOCOHQA66DFGVSSDAIAQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.757192828 (291h36m45.308663279s) [noise=true , address=145.79.227.31:4799 , piece_id=CWVNBV2KKS4CSUGALZXS3T4NSOD4WSWBFEDRGSSK3WUWSZT3EPRQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:47.214949243 (291h36m45.766419704s) [noise=true , address=45.91.116.14:4190 , piece_id=MAXQEDRM3PAP3VTYXCP4UPZKMJ7EGYW7KE3ZX2EKCZBDJGNQ24WQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:47.266695457 (291h36m45.818165918s) [noise=true , address=77.90.31.10:4519 , piece_id=3BI22NJKKK46PRFZ7MQ5VRRUMWTMCX2BU5DUVUF2XMGNWUZKBOLQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:47.321163685 (291h36m45.872634156s) [noise=true , address=77.90.31.16:4734 , piece_id=VUJDCDUQ3K5FIMDDN2Q3GI5VFXBUZJTPB5RDJYXIUY2YDYVZFWNA, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:47.321192209 (291h36m45.87266267s) [seg_pos={0 0}, upload=256808]: Segment reached optimal threshold of 13 pieces. Nov 21 11:40:47.321495266 (291h36m45.872965728s) [noise=true , address=45.91.116.10:4082 , piece_id=MRQS5HQWNHDD5QFM47OTFX5TDFZSU5JPHO7RT2Z2AFGEOALZZ44Q, seg_pos={0 0}, upload=256808]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Nov 21 11:40:47.321505776 (291h36m45.872976227s) [noise=true , address=45.91.116.10:4082 , piece_id=MRQS5HQWNHDD5QFM47OTFX5TDFZSU5JPHO7RT2Z2AFGEOALZZ44Q, seg_pos={0 0}, upload=256808]: Long tail cancellation due to optimal threshold Nov 21 11:40:47.321898474 (291h36m45.873368945s) [noise=true , address=77.90.31.19:4841 , piece_id=H4DS5OXB4H636NMNKQ3O3BPBPAIMWKD3JLF4JZBYD4HCLZYZZEQQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:44642->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:44642->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44642->77.90.31.19:4841: use of closed network connection Nov 21 11:40:47.321910357 (291h36m45.873380818s) [noise=true , address=77.90.31.19:4841 , piece_id=H4DS5OXB4H636NMNKQ3O3BPBPAIMWKD3JLF4JZBYD4HCLZYZZEQQ, seg_pos={0 0}, upload=256808]: Long tail cancellation due to optimal threshold Nov 21 11:40:47.321934122 (291h36m45.873404653s) [noise=true , address=45.91.116.7:4009 , piece_id=44SOTGVCG4ZAUNMNNGFBPJGI464X6E4XXEOOZ3W2KJYPFYIM6UHQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Nov 21 11:40:47.321943901 (291h36m45.873414362s) [noise=true , address=45.91.116.7:4009 , piece_id=44SOTGVCG4ZAUNMNNGFBPJGI464X6E4XXEOOZ3W2KJYPFYIM6UHQ, seg_pos={0 0}, upload=256808]: Long tail cancellation due to optimal threshold Nov 21 11:40:47.322083577 (291h36m45.873554048s) [seg_pos={0 0}, upload=256808]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:40:47.322197053 (291h36m45.873667514s) [upload=256808]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:40:47.322204707 (291h36m45.873675168s) [upload=256808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:40:47.613510899 (291h36m46.16498137s) [upload=256808]: Done uploading segments... Nov 21 11:40:47.613540816 (291h36m46.165011277s) [upload=256808]: Upload finished. err: <nil> Duration: 6.429961175s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......] duration: 3.46835235s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:41:41 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.468379351s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 256.133005ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:41:27 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 256.161139ms edge linksharing bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 256.287569ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:41:20 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 256.31355ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 237.802651ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:41:20 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 237.83356ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 235.811951ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:41:18 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 235.836679ms edge gateway vhost bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [............X...X.X......X.XX...X.X.X....X............X....X] duration: 30.002662088s trace: https://jaeger.dev.storj.io/trace/69e1ebd1c45373c FAIL (2025-11-21 11:41:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 3BI22NJKKK46PRFZ7MQ5VRRUMWTMCX2BU5DUVUF2XMGNWUZKBOLQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: MAXQEDRM3PAP3VTYXCP4UPZKMJ7EGYW7KE3ZX2EKCZBDJGNQ24WQ) context deadline exceeded ======LOGS====== Nov 21 11:40:44.490365952 (291h36m43.041836423s) [upload=256808]: Starting upload Nov 21 11:40:44.490456043 (291h36m43.041926504s) [upload=256808]: Done starting upload Nov 21 11:40:44.490848269 (291h36m43.04231874s) [upload=256808]: Uploading segments... Nov 21 11:40:44.490910358 (291h36m43.042380829s) [upload=256808]: Got next segment. Inline: false Nov 21 11:40:44.490949793 (291h36m43.042420274s) [upload=256808]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:40:44.490956717 (291h36m43.042427198s) [upload=256808]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:40:44.735648634 (291h36m43.287119095s) [seg_pos={0 0}, upload=256808]: Begin upload segment... Nov 21 11:40:44.735702587 (291h36m43.287173048s) [seg_pos={0 0}, upload=256808]: Done begin upload segment. Nov 21 11:40:44.735744437 (291h36m43.287214908s) [noise=true , address=77.90.31.16:4734 , piece_id=VUJDCDUQ3K5FIMDDN2Q3GI5VFXBUZJTPB5RDJYXIUY2YDYVZFWNA, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.735847072 (291h36m43.287317533s) [noise=true , address=145.79.227.8:4331 , piece_id=A26GAR4UEBCZ2DCSS3XUSV3AXDQDJ4W5L4UROJXK7GABH44BGBMA, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.736447876 (291h36m43.287918327s) [noise=true , address=145.79.227.15:4875 , piece_id=SDA2NA6L7A37XBPXCOJVZAOL4JHT3E4774XZM6VVNDZKZO3FCDHQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.736775801 (291h36m43.288246262s) [noise=true , address=77.90.31.10:4519 , piece_id=3BI22NJKKK46PRFZ7MQ5VRRUMWTMCX2BU5DUVUF2XMGNWUZKBOLQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.737089789 (291h36m43.28856025s) [noise=true , address=145.79.227.11:4730 , piece_id=FD5CGPBY3EPFNQJLF2IAZAUSZ4XSHDZVSRAUDNGOLUDFZRSTML4A, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.737427411 (291h36m43.288897892s) [noise=true , address=45.91.116.7:4009 , piece_id=44SOTGVCG4ZAUNMNNGFBPJGI464X6E4XXEOOZ3W2KJYPFYIM6UHQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.737922874 (291h36m43.289393325s) [noise=true , address=77.90.31.19:4841 , piece_id=H4DS5OXB4H636NMNKQ3O3BPBPAIMWKD3JLF4JZBYD4HCLZYZZEQQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.738266609 (291h36m43.28973706s) [noise=true , address=145.79.227.25:4618 , piece_id=QOKGVWIN2JF3E2YBGCEJR5ORVRMWUXA2K4Q6RMA4MWCJ7J5Z65PQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.738800386 (291h36m43.290270837s) [noise=true , address=45.91.116.10:4082 , piece_id=MRQS5HQWNHDD5QFM47OTFX5TDFZSU5JPHO7RT2Z2AFGEOALZZ44Q, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.739403194 (291h36m43.290873655s) [noise=true , address=145.79.227.28:4726 , piece_id=ULYH4ZQUNP7RBA7WEWFUDRP3A5BBOLNAZFKP7BS6JBDN44PSTX3A, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.739811351 (291h36m43.291281812s) [noise=true , address=145.79.227.30:4763 , piece_id=QDYPZHXOVU5BEGZJFOLZV7FKSGI37HNUVOCOHQA66DFGVSSDAIAQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.740155647 (291h36m43.291626098s) [noise=true , address=45.91.116.14:4190 , piece_id=MAXQEDRM3PAP3VTYXCP4UPZKMJ7EGYW7KE3ZX2EKCZBDJGNQ24WQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.740623147 (291h36m43.292093598s) [noise=true , address=145.79.227.11:4731 , piece_id=4EFQX5WQGOB32XDSDDVLJE6276KS6I43OAFK22H42V7ZNCIX7CNA, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.741905819 (291h36m43.29337628s) [noise=true , address=145.79.227.31:4799 , piece_id=CWVNBV2KKS4CSUGALZXS3T4NSOD4WSWBFEDRGSSK3WUWSZT3EPRQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.747098617 (291h36m43.298569078s) [noise=true , address=145.79.227.24:4581 , piece_id=OTKFUS6CNHE2HUSOPAFUB7E7MQ2ZKJE5YC3U27TFGKAVWJRM6YJQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:44.748709514 (291h36m43.300179975s) [noise=true , address=145.79.227.14:4839 , piece_id=NX6AERKM6L2RX3JN5LZQ4EBBDTL6BE2ZLL4B25JZJU2DYN2LNDOQ, seg_pos={0 0}, upload=256808]: Uploading piece... Nov 21 11:40:46.443828492 (291h36m44.995298953s) [upload=256808]: Next returned nil segment Nov 21 11:40:46.443844232 (291h36m44.995314694s) [upload=256808]: Waiting for error group managing segments... Nov 21 11:40:46.724509292 (291h36m45.275979753s) [noise=true , address=145.79.227.11:4731 , piece_id=4EFQX5WQGOB32XDSDDVLJE6276KS6I43OAFK22H42V7ZNCIX7CNA, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.731925745 (291h36m45.283396206s) [noise=true , address=145.79.227.11:4730 , piece_id=FD5CGPBY3EPFNQJLF2IAZAUSZ4XSHDZVSRAUDNGOLUDFZRSTML4A, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.732400628 (291h36m45.283871089s) [noise=true , address=145.79.227.8:4331 , piece_id=A26GAR4UEBCZ2DCSS3XUSV3AXDQDJ4W5L4UROJXK7GABH44BGBMA, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.734032606 (291h36m45.285503067s) [noise=true , address=145.79.227.24:4581 , piece_id=OTKFUS6CNHE2HUSOPAFUB7E7MQ2ZKJE5YC3U27TFGKAVWJRM6YJQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.741638760 (291h36m45.293109221s) [noise=true , address=145.79.227.25:4618 , piece_id=QOKGVWIN2JF3E2YBGCEJR5ORVRMWUXA2K4Q6RMA4MWCJ7J5Z65PQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.746594867 (291h36m45.298065328s) [noise=true , address=145.79.227.28:4726 , piece_id=ULYH4ZQUNP7RBA7WEWFUDRP3A5BBOLNAZFKP7BS6JBDN44PSTX3A, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.752194069 (291h36m45.30366453s) [noise=true , address=145.79.227.15:4875 , piece_id=SDA2NA6L7A37XBPXCOJVZAOL4JHT3E4774XZM6VVNDZKZO3FCDHQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.753504845 (291h36m45.304975306s) [noise=true , address=145.79.227.14:4839 , piece_id=NX6AERKM6L2RX3JN5LZQ4EBBDTL6BE2ZLL4B25JZJU2DYN2LNDOQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.755097948 (291h36m45.306568399s) [noise=true , address=145.79.227.30:4763 , piece_id=QDYPZHXOVU5BEGZJFOLZV7FKSGI37HNUVOCOHQA66DFGVSSDAIAQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:46.757192828 (291h36m45.308663279s) [noise=true , address=145.79.227.31:4799 , piece_id=CWVNBV2KKS4CSUGALZXS3T4NSOD4WSWBFEDRGSSK3WUWSZT3EPRQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:47.214949243 (291h36m45.766419704s) [noise=true , address=45.91.116.14:4190 , piece_id=MAXQEDRM3PAP3VTYXCP4UPZKMJ7EGYW7KE3ZX2EKCZBDJGNQ24WQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:47.266695457 (291h36m45.818165918s) [noise=true , address=77.90.31.10:4519 , piece_id=3BI22NJKKK46PRFZ7MQ5VRRUMWTMCX2BU5DUVUF2XMGNWUZKBOLQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:47.321163685 (291h36m45.872634156s) [noise=true , address=77.90.31.16:4734 , piece_id=VUJDCDUQ3K5FIMDDN2Q3GI5VFXBUZJTPB5RDJYXIUY2YDYVZFWNA, seg_pos={0 0}, upload=256808]: Done uploading piece. err: <nil> Nov 21 11:40:47.321192209 (291h36m45.87266267s) [seg_pos={0 0}, upload=256808]: Segment reached optimal threshold of 13 pieces. Nov 21 11:40:47.321495266 (291h36m45.872965728s) [noise=true , address=45.91.116.10:4082 , piece_id=MRQS5HQWNHDD5QFM47OTFX5TDFZSU5JPHO7RT2Z2AFGEOALZZ44Q, seg_pos={0 0}, upload=256808]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Nov 21 11:40:47.321505776 (291h36m45.872976227s) [noise=true , address=45.91.116.10:4082 , piece_id=MRQS5HQWNHDD5QFM47OTFX5TDFZSU5JPHO7RT2Z2AFGEOALZZ44Q, seg_pos={0 0}, upload=256808]: Long tail cancellation due to optimal threshold Nov 21 11:40:47.321898474 (291h36m45.873368945s) [noise=true , address=77.90.31.19:4841 , piece_id=H4DS5OXB4H636NMNKQ3O3BPBPAIMWKD3JLF4JZBYD4HCLZYZZEQQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:44642->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:44642->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44642->77.90.31.19:4841: use of closed network connection Nov 21 11:40:47.321910357 (291h36m45.873380818s) [noise=true , address=77.90.31.19:4841 , piece_id=H4DS5OXB4H636NMNKQ3O3BPBPAIMWKD3JLF4JZBYD4HCLZYZZEQQ, seg_pos={0 0}, upload=256808]: Long tail cancellation due to optimal threshold Nov 21 11:40:47.321934122 (291h36m45.873404653s) [noise=true , address=45.91.116.7:4009 , piece_id=44SOTGVCG4ZAUNMNNGFBPJGI464X6E4XXEOOZ3W2KJYPFYIM6UHQ, seg_pos={0 0}, upload=256808]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Nov 21 11:40:47.321943901 (291h36m45.873414362s) [noise=true , address=45.91.116.7:4009 , piece_id=44SOTGVCG4ZAUNMNNGFBPJGI464X6E4XXEOOZ3W2KJYPFYIM6UHQ, seg_pos={0 0}, upload=256808]: Long tail cancellation due to optimal threshold Nov 21 11:40:47.322083577 (291h36m45.873554048s) [seg_pos={0 0}, upload=256808]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:40:47.322197053 (291h36m45.873667514s) [upload=256808]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:40:47.322204707 (291h36m45.873675168s) [upload=256808]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:40:47.613510899 (291h36m46.16498137s) [upload=256808]: Done uploading segments... Nov 21 11:40:47.613540816 (291h36m46.165011277s) [upload=256808]: Upload finished. err: <nil> Duration: 30.002685512s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.X.......................] duration: 6.375842093s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:41:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.375873342s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 255.655448ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:40:55 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 255.685284ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 260.590775ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:40:41 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 260.617195ms edge linksharing bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......] duration: 3.361257621s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:40:38 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.361277309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.X......................] duration: 10.800615821s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:40:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 10.800643513s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 249.89623ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:40:22 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 249.936567ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 262.093235ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:40:16 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 262.136757ms edge linksharing bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....] duration: 3.337870078s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:40:15 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.337899183s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 259.277654ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:40:09 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 259.309243ms edge gateway vhost bgp eu1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 269.756788ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:40:04 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 269.788607ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 238.864553ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:39:56 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 238.88928ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.............X...X.X......X.XX...X.X.X....X............X....] duration: 5.716414499s trace: https://jaeger.dev.storj.io/trace/78da79dc91888b5e FAIL (2025-11-21 11:39:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 11:36:38.653343650 (291h32m37.204814111s) [upload=256743]: Starting upload Nov 21 11:36:38.653440093 (291h32m37.204910554s) [upload=256743]: Done starting upload Nov 21 11:36:38.653574619 (291h32m37.20504509s) [upload=256743]: Uploading segments... Nov 21 11:36:38.653739233 (291h32m37.205209704s) [upload=256743]: Got next segment. Inline: false Nov 21 11:36:38.653853891 (291h32m37.205324352s) [upload=256743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:36:38.653861345 (291h32m37.205331806s) [upload=256743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:36:38.902234785 (291h32m37.453705246s) [seg_pos={0 0}, upload=256743]: Begin upload segment... Nov 21 11:36:38.902298005 (291h32m37.453768466s) [seg_pos={0 0}, upload=256743]: Done begin upload segment. Nov 21 11:36:38.902338562 (291h32m37.453809023s) [noise=true , address=145.79.227.28:4726 , piece_id=AL7GMARXLTYWZBOKXTV3AKVAM7AUSUK7EY54JMQ2QX5ZIRW3ZZ7Q, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.902584601 (291h32m37.454055062s) [noise=true , address=45.91.116.5:4298 , piece_id=XBEXRPZG6W4K7OETP65Q6LEGOTHHP3CTBYEAYGBDWWZLXXXRWTSA, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.903096405 (291h32m37.454566866s) [noise=true , address=145.79.227.14:4837 , piece_id=SUM6K2L773TXHSJDUTZTW4A5GHEGUDW2UJLOPWKXVZQAGXCFKDEA, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.903567612 (291h32m37.455038073s) [noise=true , address=145.79.227.16:4910 , piece_id=UI67ANC6E5OKSGIZMD6INC6HO3C6JIXLIJSTCXGFGVK3PVQLSFOQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.904689188 (291h32m37.456159639s) [noise=true , address=145.79.227.27:4690 , piece_id=2RDNP66XUPVXBW3INKOB7EGYFWUYTQ27SJP7DNTKOWVJXZRZHOUQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.904882606 (291h32m37.456353068s) [noise=true , address=45.91.116.7:4011 , piece_id=Z2GUJE6YC64HDVG5N56MA34G7DBPZRF2SZLBSLNHUICHJYASLGOQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.905207204 (291h32m37.456677665s) [noise=true , address=145.79.227.30:4763 , piece_id=AUQDV6OH2YRFYFTY4LM75SV27PZWWLKBG2NTMALYYP2YE3R2Z5RQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.906315445 (291h32m37.457785906s) [noise=true , address=145.79.227.23:4545 , piece_id=WXD27B5H2P6TEN6CUD2PEPQLQP3HR3EMRFL7DGUPGKOMYMPXQBUQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.906486130 (291h32m37.457956591s) [noise=true , address=145.79.227.15:4875 , piece_id=ANFL6R6GWBQWL5XAX547H7BJOH355GUVCCWGYH73QGRIIXBO5JIA, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.907440116 (291h32m37.458910577s) [noise=true , address=145.79.227.26:4653 , piece_id=MWCUKH73GCYIOBWLRCKVGMTTWKKDPP2OTUPBDGG5MSLGQPORIQUQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.908672623 (291h32m37.460143085s) [noise=true , address=45.91.116.14:4190 , piece_id=KSG3ISJIQCM73VN5AZU5NGYNQCZ4L7HMXMRSWHJZFWX6IG5BL5KQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.909517041 (291h32m37.460987502s) [noise=true , address=45.91.116.4:4940 , piece_id=YTOJWH7HBZS75VHMSSRTSW4IO7KZ5ZXOZQRXLEHPUMUY5DVOBCMQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.910476377 (291h32m37.461946828s) [noise=true , address=77.90.31.21:4879 , piece_id=VHKHNVDJM35FTBF34YJFSKMMGJIH7CJJ7O6CEU2I4XAFYWISRQYQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.912189600 (291h32m37.463660061s) [noise=true , address=145.79.227.25:4618 , piece_id=SLOZRK3PNHVJBUCJDUXASP2HX4RW7MR3QI2XYIXZ4KMDKBHQW3QQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.913314422 (291h32m37.464784883s) [noise=true , address=45.91.116.10:4082 , piece_id=4YJX6FOJC6PTZVYP4V63LPEND3ZYIJYFWBOPI2N3332CLBSYMIYA, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.924075122 (291h32m37.475545583s) [noise=true , address=45.91.116.8:4047 , piece_id=BCBSXIAF2HGANZVCQHAZCSN77I4HPEEZ3JJSALTMCQKCIKEVMQOQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:40.529316692 (291h32m39.080787153s) [upload=256743]: Next returned nil segment Nov 21 11:36:40.529335277 (291h32m39.080805749s) [upload=256743]: Waiting for error group managing segments... Nov 21 11:36:40.905438387 (291h32m39.456908849s) [noise=true , address=145.79.227.16:4910 , piece_id=UI67ANC6E5OKSGIZMD6INC6HO3C6JIXLIJSTCXGFGVK3PVQLSFOQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.907956983 (291h32m39.459427444s) [noise=true , address=145.79.227.14:4837 , piece_id=SUM6K2L773TXHSJDUTZTW4A5GHEGUDW2UJLOPWKXVZQAGXCFKDEA, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.909262139 (291h32m39.46073262s) [noise=true , address=145.79.227.15:4875 , piece_id=ANFL6R6GWBQWL5XAX547H7BJOH355GUVCCWGYH73QGRIIXBO5JIA, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.913758611 (291h32m39.465229082s) [noise=true , address=145.79.227.30:4763 , piece_id=AUQDV6OH2YRFYFTY4LM75SV27PZWWLKBG2NTMALYYP2YE3R2Z5RQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.915622750 (291h32m39.467093201s) [noise=true , address=145.79.227.28:4726 , piece_id=AL7GMARXLTYWZBOKXTV3AKVAM7AUSUK7EY54JMQ2QX5ZIRW3ZZ7Q, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.916487967 (291h32m39.467958438s) [noise=true , address=145.79.227.23:4545 , piece_id=WXD27B5H2P6TEN6CUD2PEPQLQP3HR3EMRFL7DGUPGKOMYMPXQBUQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.921414589 (291h32m39.47288505s) [noise=true , address=145.79.227.26:4653 , piece_id=MWCUKH73GCYIOBWLRCKVGMTTWKKDPP2OTUPBDGG5MSLGQPORIQUQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.924455340 (291h32m39.475925801s) [noise=true , address=145.79.227.25:4618 , piece_id=SLOZRK3PNHVJBUCJDUXASP2HX4RW7MR3QI2XYIXZ4KMDKBHQW3QQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.930004646 (291h32m39.481475107s) [noise=true , address=145.79.227.27:4690 , piece_id=2RDNP66XUPVXBW3INKOB7EGYFWUYTQ27SJP7DNTKOWVJXZRZHOUQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.470609128 (291h32m40.022079589s) [noise=true , address=45.91.116.14:4190 , piece_id=KSG3ISJIQCM73VN5AZU5NGYNQCZ4L7HMXMRSWHJZFWX6IG5BL5KQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.483302128 (291h32m40.034772589s) [noise=true , address=45.91.116.5:4298 , piece_id=XBEXRPZG6W4K7OETP65Q6LEGOTHHP3CTBYEAYGBDWWZLXXXRWTSA, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.484869332 (291h32m40.036339793s) [noise=true , address=45.91.116.4:4940 , piece_id=YTOJWH7HBZS75VHMSSRTSW4IO7KZ5ZXOZQRXLEHPUMUY5DVOBCMQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.537287535 (291h32m40.088757996s) [noise=true , address=45.91.116.10:4082 , piece_id=4YJX6FOJC6PTZVYP4V63LPEND3ZYIJYFWBOPI2N3332CLBSYMIYA, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.537314767 (291h32m40.088785228s) [seg_pos={0 0}, upload=256743]: Segment reached optimal threshold of 13 pieces. Nov 21 11:36:41.537539776 (291h32m40.089010237s) [noise=true , address=45.91.116.8:4047 , piece_id=BCBSXIAF2HGANZVCQHAZCSN77I4HPEEZ3JJSALTMCQKCIKEVMQOQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Nov 21 11:36:41.537548793 (291h32m40.089019254s) [noise=true , address=45.91.116.8:4047 , piece_id=BCBSXIAF2HGANZVCQHAZCSN77I4HPEEZ3JJSALTMCQKCIKEVMQOQ, seg_pos={0 0}, upload=256743]: Long tail cancellation due to optimal threshold Nov 21 11:36:41.537909780 (291h32m40.089380251s) [noise=true , address=45.91.116.7:4011 , piece_id=Z2GUJE6YC64HDVG5N56MA34G7DBPZRF2SZLBSLNHUICHJYASLGOQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Nov 21 11:36:41.537918788 (291h32m40.089389249s) [noise=true , address=45.91.116.7:4011 , piece_id=Z2GUJE6YC64HDVG5N56MA34G7DBPZRF2SZLBSLNHUICHJYASLGOQ, seg_pos={0 0}, upload=256743]: Long tail cancellation due to optimal threshold Nov 21 11:36:41.538081828 (291h32m40.089552289s) [noise=true , address=77.90.31.21:4879 , piece_id=VHKHNVDJM35FTBF34YJFSKMMGJIH7CJJ7O6CEU2I4XAFYWISRQYQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Nov 21 11:36:41.538090885 (291h32m40.089561346s) [noise=true , address=77.90.31.21:4879 , piece_id=VHKHNVDJM35FTBF34YJFSKMMGJIH7CJJ7O6CEU2I4XAFYWISRQYQ, seg_pos={0 0}, upload=256743]: Long tail cancellation due to optimal threshold Nov 21 11:36:41.538285286 (291h32m40.089755747s) [seg_pos={0 0}, upload=256743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:36:41.538383423 (291h32m40.089853894s) [upload=256743]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:36:41.538390566 (291h32m40.089861027s) [upload=256743]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:36:41.833739024 (291h32m40.385209485s) [upload=256743]: Done uploading segments... Nov 21 11:36:41.833763751 (291h32m40.385234202s) [upload=256743]: Upload finished. err: <nil> Duration: 5.716442744s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 259.445944ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:39:52 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 259.466462ms edge linksharing bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 248.947183ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:39:36 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 248.976959ms edge gateway vhost bgp eu1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 257.664521ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:39:35 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 257.693898ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 255.518877ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:39:30 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 255.544855ms edge linksharing bgp us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.X.....................] duration: 6.937675577s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:39:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.937702869s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 251.425642ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:39:16 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 251.461941ms edge linksharing bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 3.450194354s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:39:12 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.450223229s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 258.030188ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:39:07 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 258.062258ms edge linksharing bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..............X...X.X......X.XX...X.X.X....X............X...] duration: 6.308235445s trace: https://jaeger.dev.storj.io/trace/78da79dc91888b5e FAIL (2025-11-21 11:39:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 11:36:38.653343650 (291h32m37.204814111s) [upload=256743]: Starting upload Nov 21 11:36:38.653440093 (291h32m37.204910554s) [upload=256743]: Done starting upload Nov 21 11:36:38.653574619 (291h32m37.20504509s) [upload=256743]: Uploading segments... Nov 21 11:36:38.653739233 (291h32m37.205209704s) [upload=256743]: Got next segment. Inline: false Nov 21 11:36:38.653853891 (291h32m37.205324352s) [upload=256743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:36:38.653861345 (291h32m37.205331806s) [upload=256743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:36:38.902234785 (291h32m37.453705246s) [seg_pos={0 0}, upload=256743]: Begin upload segment... Nov 21 11:36:38.902298005 (291h32m37.453768466s) [seg_pos={0 0}, upload=256743]: Done begin upload segment. Nov 21 11:36:38.902338562 (291h32m37.453809023s) [noise=true , address=145.79.227.28:4726 , piece_id=AL7GMARXLTYWZBOKXTV3AKVAM7AUSUK7EY54JMQ2QX5ZIRW3ZZ7Q, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.902584601 (291h32m37.454055062s) [noise=true , address=45.91.116.5:4298 , piece_id=XBEXRPZG6W4K7OETP65Q6LEGOTHHP3CTBYEAYGBDWWZLXXXRWTSA, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.903096405 (291h32m37.454566866s) [noise=true , address=145.79.227.14:4837 , piece_id=SUM6K2L773TXHSJDUTZTW4A5GHEGUDW2UJLOPWKXVZQAGXCFKDEA, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.903567612 (291h32m37.455038073s) [noise=true , address=145.79.227.16:4910 , piece_id=UI67ANC6E5OKSGIZMD6INC6HO3C6JIXLIJSTCXGFGVK3PVQLSFOQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.904689188 (291h32m37.456159639s) [noise=true , address=145.79.227.27:4690 , piece_id=2RDNP66XUPVXBW3INKOB7EGYFWUYTQ27SJP7DNTKOWVJXZRZHOUQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.904882606 (291h32m37.456353068s) [noise=true , address=45.91.116.7:4011 , piece_id=Z2GUJE6YC64HDVG5N56MA34G7DBPZRF2SZLBSLNHUICHJYASLGOQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.905207204 (291h32m37.456677665s) [noise=true , address=145.79.227.30:4763 , piece_id=AUQDV6OH2YRFYFTY4LM75SV27PZWWLKBG2NTMALYYP2YE3R2Z5RQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.906315445 (291h32m37.457785906s) [noise=true , address=145.79.227.23:4545 , piece_id=WXD27B5H2P6TEN6CUD2PEPQLQP3HR3EMRFL7DGUPGKOMYMPXQBUQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.906486130 (291h32m37.457956591s) [noise=true , address=145.79.227.15:4875 , piece_id=ANFL6R6GWBQWL5XAX547H7BJOH355GUVCCWGYH73QGRIIXBO5JIA, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.907440116 (291h32m37.458910577s) [noise=true , address=145.79.227.26:4653 , piece_id=MWCUKH73GCYIOBWLRCKVGMTTWKKDPP2OTUPBDGG5MSLGQPORIQUQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.908672623 (291h32m37.460143085s) [noise=true , address=45.91.116.14:4190 , piece_id=KSG3ISJIQCM73VN5AZU5NGYNQCZ4L7HMXMRSWHJZFWX6IG5BL5KQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.909517041 (291h32m37.460987502s) [noise=true , address=45.91.116.4:4940 , piece_id=YTOJWH7HBZS75VHMSSRTSW4IO7KZ5ZXOZQRXLEHPUMUY5DVOBCMQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.910476377 (291h32m37.461946828s) [noise=true , address=77.90.31.21:4879 , piece_id=VHKHNVDJM35FTBF34YJFSKMMGJIH7CJJ7O6CEU2I4XAFYWISRQYQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.912189600 (291h32m37.463660061s) [noise=true , address=145.79.227.25:4618 , piece_id=SLOZRK3PNHVJBUCJDUXASP2HX4RW7MR3QI2XYIXZ4KMDKBHQW3QQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.913314422 (291h32m37.464784883s) [noise=true , address=45.91.116.10:4082 , piece_id=4YJX6FOJC6PTZVYP4V63LPEND3ZYIJYFWBOPI2N3332CLBSYMIYA, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.924075122 (291h32m37.475545583s) [noise=true , address=45.91.116.8:4047 , piece_id=BCBSXIAF2HGANZVCQHAZCSN77I4HPEEZ3JJSALTMCQKCIKEVMQOQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:40.529316692 (291h32m39.080787153s) [upload=256743]: Next returned nil segment Nov 21 11:36:40.529335277 (291h32m39.080805749s) [upload=256743]: Waiting for error group managing segments... Nov 21 11:36:40.905438387 (291h32m39.456908849s) [noise=true , address=145.79.227.16:4910 , piece_id=UI67ANC6E5OKSGIZMD6INC6HO3C6JIXLIJSTCXGFGVK3PVQLSFOQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.907956983 (291h32m39.459427444s) [noise=true , address=145.79.227.14:4837 , piece_id=SUM6K2L773TXHSJDUTZTW4A5GHEGUDW2UJLOPWKXVZQAGXCFKDEA, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.909262139 (291h32m39.46073262s) [noise=true , address=145.79.227.15:4875 , piece_id=ANFL6R6GWBQWL5XAX547H7BJOH355GUVCCWGYH73QGRIIXBO5JIA, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.913758611 (291h32m39.465229082s) [noise=true , address=145.79.227.30:4763 , piece_id=AUQDV6OH2YRFYFTY4LM75SV27PZWWLKBG2NTMALYYP2YE3R2Z5RQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.915622750 (291h32m39.467093201s) [noise=true , address=145.79.227.28:4726 , piece_id=AL7GMARXLTYWZBOKXTV3AKVAM7AUSUK7EY54JMQ2QX5ZIRW3ZZ7Q, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.916487967 (291h32m39.467958438s) [noise=true , address=145.79.227.23:4545 , piece_id=WXD27B5H2P6TEN6CUD2PEPQLQP3HR3EMRFL7DGUPGKOMYMPXQBUQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.921414589 (291h32m39.47288505s) [noise=true , address=145.79.227.26:4653 , piece_id=MWCUKH73GCYIOBWLRCKVGMTTWKKDPP2OTUPBDGG5MSLGQPORIQUQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.924455340 (291h32m39.475925801s) [noise=true , address=145.79.227.25:4618 , piece_id=SLOZRK3PNHVJBUCJDUXASP2HX4RW7MR3QI2XYIXZ4KMDKBHQW3QQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.930004646 (291h32m39.481475107s) [noise=true , address=145.79.227.27:4690 , piece_id=2RDNP66XUPVXBW3INKOB7EGYFWUYTQ27SJP7DNTKOWVJXZRZHOUQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.470609128 (291h32m40.022079589s) [noise=true , address=45.91.116.14:4190 , piece_id=KSG3ISJIQCM73VN5AZU5NGYNQCZ4L7HMXMRSWHJZFWX6IG5BL5KQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.483302128 (291h32m40.034772589s) [noise=true , address=45.91.116.5:4298 , piece_id=XBEXRPZG6W4K7OETP65Q6LEGOTHHP3CTBYEAYGBDWWZLXXXRWTSA, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.484869332 (291h32m40.036339793s) [noise=true , address=45.91.116.4:4940 , piece_id=YTOJWH7HBZS75VHMSSRTSW4IO7KZ5ZXOZQRXLEHPUMUY5DVOBCMQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.537287535 (291h32m40.088757996s) [noise=true , address=45.91.116.10:4082 , piece_id=4YJX6FOJC6PTZVYP4V63LPEND3ZYIJYFWBOPI2N3332CLBSYMIYA, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.537314767 (291h32m40.088785228s) [seg_pos={0 0}, upload=256743]: Segment reached optimal threshold of 13 pieces. Nov 21 11:36:41.537539776 (291h32m40.089010237s) [noise=true , address=45.91.116.8:4047 , piece_id=BCBSXIAF2HGANZVCQHAZCSN77I4HPEEZ3JJSALTMCQKCIKEVMQOQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Nov 21 11:36:41.537548793 (291h32m40.089019254s) [noise=true , address=45.91.116.8:4047 , piece_id=BCBSXIAF2HGANZVCQHAZCSN77I4HPEEZ3JJSALTMCQKCIKEVMQOQ, seg_pos={0 0}, upload=256743]: Long tail cancellation due to optimal threshold Nov 21 11:36:41.537909780 (291h32m40.089380251s) [noise=true , address=45.91.116.7:4011 , piece_id=Z2GUJE6YC64HDVG5N56MA34G7DBPZRF2SZLBSLNHUICHJYASLGOQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Nov 21 11:36:41.537918788 (291h32m40.089389249s) [noise=true , address=45.91.116.7:4011 , piece_id=Z2GUJE6YC64HDVG5N56MA34G7DBPZRF2SZLBSLNHUICHJYASLGOQ, seg_pos={0 0}, upload=256743]: Long tail cancellation due to optimal threshold Nov 21 11:36:41.538081828 (291h32m40.089552289s) [noise=true , address=77.90.31.21:4879 , piece_id=VHKHNVDJM35FTBF34YJFSKMMGJIH7CJJ7O6CEU2I4XAFYWISRQYQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Nov 21 11:36:41.538090885 (291h32m40.089561346s) [noise=true , address=77.90.31.21:4879 , piece_id=VHKHNVDJM35FTBF34YJFSKMMGJIH7CJJ7O6CEU2I4XAFYWISRQYQ, seg_pos={0 0}, upload=256743]: Long tail cancellation due to optimal threshold Nov 21 11:36:41.538285286 (291h32m40.089755747s) [seg_pos={0 0}, upload=256743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:36:41.538383423 (291h32m40.089853894s) [upload=256743]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:36:41.538390566 (291h32m40.089861027s) [upload=256743]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:36:41.833739024 (291h32m40.385209485s) [upload=256743]: Done uploading segments... Nov 21 11:36:41.833763751 (291h32m40.385234202s) [upload=256743]: Upload finished. err: <nil> Duration: 6.308259082s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.X....................] duration: 5.738042242s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:38:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.738071547s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 256.722908ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:38:53 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 256.758747ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 378.927229ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:38:40 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 378.959148ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.X...................] duration: 5.796694579s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:38:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.796723985s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 249.377522ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:38:34 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 249.410034ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 248.145826ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:38:34 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 248.171424ms edge gateway vhost bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...............X...X.X......X.XX...X.X.X....X............X..] duration: 9.853031169s trace: https://jaeger.dev.storj.io/trace/78da79dc91888b5e FAIL (2025-11-21 11:38:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 21 11:36:38.653343650 (291h32m37.204814111s) [upload=256743]: Starting upload Nov 21 11:36:38.653440093 (291h32m37.204910554s) [upload=256743]: Done starting upload Nov 21 11:36:38.653574619 (291h32m37.20504509s) [upload=256743]: Uploading segments... Nov 21 11:36:38.653739233 (291h32m37.205209704s) [upload=256743]: Got next segment. Inline: false Nov 21 11:36:38.653853891 (291h32m37.205324352s) [upload=256743]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:36:38.653861345 (291h32m37.205331806s) [upload=256743]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:36:38.902234785 (291h32m37.453705246s) [seg_pos={0 0}, upload=256743]: Begin upload segment... Nov 21 11:36:38.902298005 (291h32m37.453768466s) [seg_pos={0 0}, upload=256743]: Done begin upload segment. Nov 21 11:36:38.902338562 (291h32m37.453809023s) [noise=true , address=145.79.227.28:4726 , piece_id=AL7GMARXLTYWZBOKXTV3AKVAM7AUSUK7EY54JMQ2QX5ZIRW3ZZ7Q, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.902584601 (291h32m37.454055062s) [noise=true , address=45.91.116.5:4298 , piece_id=XBEXRPZG6W4K7OETP65Q6LEGOTHHP3CTBYEAYGBDWWZLXXXRWTSA, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.903096405 (291h32m37.454566866s) [noise=true , address=145.79.227.14:4837 , piece_id=SUM6K2L773TXHSJDUTZTW4A5GHEGUDW2UJLOPWKXVZQAGXCFKDEA, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.903567612 (291h32m37.455038073s) [noise=true , address=145.79.227.16:4910 , piece_id=UI67ANC6E5OKSGIZMD6INC6HO3C6JIXLIJSTCXGFGVK3PVQLSFOQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.904689188 (291h32m37.456159639s) [noise=true , address=145.79.227.27:4690 , piece_id=2RDNP66XUPVXBW3INKOB7EGYFWUYTQ27SJP7DNTKOWVJXZRZHOUQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.904882606 (291h32m37.456353068s) [noise=true , address=45.91.116.7:4011 , piece_id=Z2GUJE6YC64HDVG5N56MA34G7DBPZRF2SZLBSLNHUICHJYASLGOQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.905207204 (291h32m37.456677665s) [noise=true , address=145.79.227.30:4763 , piece_id=AUQDV6OH2YRFYFTY4LM75SV27PZWWLKBG2NTMALYYP2YE3R2Z5RQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.906315445 (291h32m37.457785906s) [noise=true , address=145.79.227.23:4545 , piece_id=WXD27B5H2P6TEN6CUD2PEPQLQP3HR3EMRFL7DGUPGKOMYMPXQBUQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.906486130 (291h32m37.457956591s) [noise=true , address=145.79.227.15:4875 , piece_id=ANFL6R6GWBQWL5XAX547H7BJOH355GUVCCWGYH73QGRIIXBO5JIA, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.907440116 (291h32m37.458910577s) [noise=true , address=145.79.227.26:4653 , piece_id=MWCUKH73GCYIOBWLRCKVGMTTWKKDPP2OTUPBDGG5MSLGQPORIQUQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.908672623 (291h32m37.460143085s) [noise=true , address=45.91.116.14:4190 , piece_id=KSG3ISJIQCM73VN5AZU5NGYNQCZ4L7HMXMRSWHJZFWX6IG5BL5KQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.909517041 (291h32m37.460987502s) [noise=true , address=45.91.116.4:4940 , piece_id=YTOJWH7HBZS75VHMSSRTSW4IO7KZ5ZXOZQRXLEHPUMUY5DVOBCMQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.910476377 (291h32m37.461946828s) [noise=true , address=77.90.31.21:4879 , piece_id=VHKHNVDJM35FTBF34YJFSKMMGJIH7CJJ7O6CEU2I4XAFYWISRQYQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.912189600 (291h32m37.463660061s) [noise=true , address=145.79.227.25:4618 , piece_id=SLOZRK3PNHVJBUCJDUXASP2HX4RW7MR3QI2XYIXZ4KMDKBHQW3QQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.913314422 (291h32m37.464784883s) [noise=true , address=45.91.116.10:4082 , piece_id=4YJX6FOJC6PTZVYP4V63LPEND3ZYIJYFWBOPI2N3332CLBSYMIYA, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:38.924075122 (291h32m37.475545583s) [noise=true , address=45.91.116.8:4047 , piece_id=BCBSXIAF2HGANZVCQHAZCSN77I4HPEEZ3JJSALTMCQKCIKEVMQOQ, seg_pos={0 0}, upload=256743]: Uploading piece... Nov 21 11:36:40.529316692 (291h32m39.080787153s) [upload=256743]: Next returned nil segment Nov 21 11:36:40.529335277 (291h32m39.080805749s) [upload=256743]: Waiting for error group managing segments... Nov 21 11:36:40.905438387 (291h32m39.456908849s) [noise=true , address=145.79.227.16:4910 , piece_id=UI67ANC6E5OKSGIZMD6INC6HO3C6JIXLIJSTCXGFGVK3PVQLSFOQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.907956983 (291h32m39.459427444s) [noise=true , address=145.79.227.14:4837 , piece_id=SUM6K2L773TXHSJDUTZTW4A5GHEGUDW2UJLOPWKXVZQAGXCFKDEA, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.909262139 (291h32m39.46073262s) [noise=true , address=145.79.227.15:4875 , piece_id=ANFL6R6GWBQWL5XAX547H7BJOH355GUVCCWGYH73QGRIIXBO5JIA, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.913758611 (291h32m39.465229082s) [noise=true , address=145.79.227.30:4763 , piece_id=AUQDV6OH2YRFYFTY4LM75SV27PZWWLKBG2NTMALYYP2YE3R2Z5RQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.915622750 (291h32m39.467093201s) [noise=true , address=145.79.227.28:4726 , piece_id=AL7GMARXLTYWZBOKXTV3AKVAM7AUSUK7EY54JMQ2QX5ZIRW3ZZ7Q, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.916487967 (291h32m39.467958438s) [noise=true , address=145.79.227.23:4545 , piece_id=WXD27B5H2P6TEN6CUD2PEPQLQP3HR3EMRFL7DGUPGKOMYMPXQBUQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.921414589 (291h32m39.47288505s) [noise=true , address=145.79.227.26:4653 , piece_id=MWCUKH73GCYIOBWLRCKVGMTTWKKDPP2OTUPBDGG5MSLGQPORIQUQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.924455340 (291h32m39.475925801s) [noise=true , address=145.79.227.25:4618 , piece_id=SLOZRK3PNHVJBUCJDUXASP2HX4RW7MR3QI2XYIXZ4KMDKBHQW3QQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:40.930004646 (291h32m39.481475107s) [noise=true , address=145.79.227.27:4690 , piece_id=2RDNP66XUPVXBW3INKOB7EGYFWUYTQ27SJP7DNTKOWVJXZRZHOUQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.470609128 (291h32m40.022079589s) [noise=true , address=45.91.116.14:4190 , piece_id=KSG3ISJIQCM73VN5AZU5NGYNQCZ4L7HMXMRSWHJZFWX6IG5BL5KQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.483302128 (291h32m40.034772589s) [noise=true , address=45.91.116.5:4298 , piece_id=XBEXRPZG6W4K7OETP65Q6LEGOTHHP3CTBYEAYGBDWWZLXXXRWTSA, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.484869332 (291h32m40.036339793s) [noise=true , address=45.91.116.4:4940 , piece_id=YTOJWH7HBZS75VHMSSRTSW4IO7KZ5ZXOZQRXLEHPUMUY5DVOBCMQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.537287535 (291h32m40.088757996s) [noise=true , address=45.91.116.10:4082 , piece_id=4YJX6FOJC6PTZVYP4V63LPEND3ZYIJYFWBOPI2N3332CLBSYMIYA, seg_pos={0 0}, upload=256743]: Done uploading piece. err: <nil> Nov 21 11:36:41.537314767 (291h32m40.088785228s) [seg_pos={0 0}, upload=256743]: Segment reached optimal threshold of 13 pieces. Nov 21 11:36:41.537539776 (291h32m40.089010237s) [noise=true , address=45.91.116.8:4047 , piece_id=BCBSXIAF2HGANZVCQHAZCSN77I4HPEEZ3JJSALTMCQKCIKEVMQOQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Nov 21 11:36:41.537548793 (291h32m40.089019254s) [noise=true , address=45.91.116.8:4047 , piece_id=BCBSXIAF2HGANZVCQHAZCSN77I4HPEEZ3JJSALTMCQKCIKEVMQOQ, seg_pos={0 0}, upload=256743]: Long tail cancellation due to optimal threshold Nov 21 11:36:41.537909780 (291h32m40.089380251s) [noise=true , address=45.91.116.7:4011 , piece_id=Z2GUJE6YC64HDVG5N56MA34G7DBPZRF2SZLBSLNHUICHJYASLGOQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Nov 21 11:36:41.537918788 (291h32m40.089389249s) [noise=true , address=45.91.116.7:4011 , piece_id=Z2GUJE6YC64HDVG5N56MA34G7DBPZRF2SZLBSLNHUICHJYASLGOQ, seg_pos={0 0}, upload=256743]: Long tail cancellation due to optimal threshold Nov 21 11:36:41.538081828 (291h32m40.089552289s) [noise=true , address=77.90.31.21:4879 , piece_id=VHKHNVDJM35FTBF34YJFSKMMGJIH7CJJ7O6CEU2I4XAFYWISRQYQ, seg_pos={0 0}, upload=256743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: expected piece hash; context canceled; EOF Nov 21 11:36:41.538090885 (291h32m40.089561346s) [noise=true , address=77.90.31.21:4879 , piece_id=VHKHNVDJM35FTBF34YJFSKMMGJIH7CJJ7O6CEU2I4XAFYWISRQYQ, seg_pos={0 0}, upload=256743]: Long tail cancellation due to optimal threshold Nov 21 11:36:41.538285286 (291h32m40.089755747s) [seg_pos={0 0}, upload=256743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:36:41.538383423 (291h32m40.089853894s) [upload=256743]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:36:41.538390566 (291h32m40.089861027s) [upload=256743]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:36:41.833739024 (291h32m40.385209485s) [upload=256743]: Done uploading segments... Nov 21 11:36:41.833763751 (291h32m40.385234202s) [upload=256743]: Upload finished. err: <nil> Duration: 9.853061507s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 3.389168187s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:38:26 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.389194919s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 258.220609ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:38:03 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 258.327234ms edge linksharing bgp us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.X..................] duration: 5.907461999s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:37:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.907491534s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 247.82816ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:37:52 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 247.856916ms edge gateway vhost bgp eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.X.................] duration: 6.756587342s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:37:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.756622339s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 3.407210483s trace: https://jaeger.dev.storj.io/trace/6af15a3a67b655d5 FAIL (2025-11-21 11:37:35 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.407234792s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)