Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........................X...X.X......X.XX...X.X.X....X......] duration: 6.448735734s trace: https://jaeger.dev.storj.io/trace/7e7a6aa5cbba947b FAIL (2025-11-21 11:31:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ov 21 11:27:17.125486410 (291h23m15.676956861s) [upload=256610]: Starting upload Nov 21 11:27:17.125579928 (291h23m15.677050399s) [upload=256610]: Done starting upload Nov 21 11:27:17.125690539 (291h23m15.677161s) [upload=256610]: Uploading segments... Nov 21 11:27:17.126036017 (291h23m15.677506468s) [upload=256610]: Got next segment. Inline: false Nov 21 11:27:17.126069811 (291h23m15.677540262s) [upload=256610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:27:17.126074891 (291h23m15.677545352s) [upload=256610]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:27:17.373868957 (291h23m15.925339418s) [seg_pos={0 0}, upload=256610]: Begin upload segment... Nov 21 11:27:17.373965912 (291h23m15.925436373s) [seg_pos={0 0}, upload=256610]: Done begin upload segment. Nov 21 11:27:17.373988064 (291h23m15.925458525s) [noise=true , address=77.90.31.18:4805 , piece_id=3E62LR5OVVTS3O7CFDRDABMILFBBVE6PS74CXRMTPGC4CVAXNW2A, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.374015958 (291h23m15.925486419s) [noise=true , address=145.79.227.23:4545 , piece_id=6MBV4336KPSHR3S67T333ZQCLXX5QBXOYAEBAUG76CVDU2CVDJXA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.374732552 (291h23m15.926203013s) [noise=true , address=45.91.116.12:4225 , piece_id=3OUXQAWMEGWO26XBUBRNAEHXV2DP5EUP5UUKNVRZHHPMASJ57OXQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375180444 (291h23m15.926650906s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375516435 (291h23m15.926986886s) [noise=true , address=145.79.227.24:4581 , piece_id=VVU2M3HLNS4A46SSIQRCOR6WV6J5FUHREINR3TC2H4RJBV4GZNFQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375888683 (291h23m15.927359144s) [noise=true , address=145.79.227.16:4909 , piece_id=5JKO5HSDEUTJN4TK6CUZ273EYNR6A6Y3625TUUOS3VKHMAPWDHFQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.376543971 (291h23m15.928014432s) [noise=true , address=145.79.227.21:4473 , piece_id=ILRSOD2EDSJXRMVZ7NITZOAZTRQQ5UUW3GKB3QB7DQASVRQVC27A, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.376981635 (291h23m15.928452096s) [noise=true , address=145.79.227.11:4731 , piece_id=CPBGYXE7CIJRVCLWCCMTL22SL2ZSC62KRQINNNACO4MYI4YLRGGQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.377562520 (291h23m15.929032981s) [noise=true , address=145.79.227.31:4799 , piece_id=CHOH62M4JKZGHNHR76M2WUKPE7G772IHGQG7UHYU5PVAYEKC4VPQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.378610416 (291h23m15.930080877s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWQ2CNVJCR5FKJ3OA5MKPH42O27BKLSTIJJIPKEQSDM3EJHNGRQA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.378974038 (291h23m15.930444499s) [noise=true , address=145.79.227.28:4726 , piece_id=72XULSSYQMZQAXFCRQFMXOBRCOE2LILOV6FPDORRPMWI6GVBVGHQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.379921312 (291h23m15.931391773s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.380443847 (291h23m15.931914308s) [noise=true , address=77.90.31.12:4589 , piece_id=S4JENPNCKPBMMRSMBD7AFKWWT6VI2DWM4YWE4O7IPTIZYEBZFOMA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.381260582 (291h23m15.932731044s) [noise=true , address=145.79.227.10:4402 , piece_id=FDDET6WXFVXZJIG2YENY3RVK3SC2LEOSRCR25VLYINEWW5YFJHYQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.381929115 (291h23m15.933399576s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.383439992 (291h23m15.934910473s) [noise=true , address=45.91.116.4:4940 , piece_id=OXPGVCFNLUECUPAETAAO6N42JULU6PJQJTXZU4PP6XFA3ENGUFYA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:18.997259485 (291h23m17.548729936s) [upload=256610]: Next returned nil segment Nov 21 11:27:18.997276468 (291h23m17.548746929s) [upload=256610]: Waiting for error group managing segments... Nov 21 11:27:19.376515178 (291h23m17.927985629s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWQ2CNVJCR5FKJ3OA5MKPH42O27BKLSTIJJIPKEQSDM3EJHNGRQA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.376615770 (291h23m17.928086231s) [noise=true , address=145.79.227.21:4473 , piece_id=ILRSOD2EDSJXRMVZ7NITZOAZTRQQ5UUW3GKB3QB7DQASVRQVC27A, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.380558629 (291h23m17.93202909s) [noise=true , address=145.79.227.11:4731 , piece_id=CPBGYXE7CIJRVCLWCCMTL22SL2ZSC62KRQINNNACO4MYI4YLRGGQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.386618928 (291h23m17.93808942s) [noise=true , address=145.79.227.23:4545 , piece_id=6MBV4336KPSHR3S67T333ZQCLXX5QBXOYAEBAUG76CVDU2CVDJXA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.388538574 (291h23m17.940009035s) [noise=true , address=145.79.227.24:4581 , piece_id=VVU2M3HLNS4A46SSIQRCOR6WV6J5FUHREINR3TC2H4RJBV4GZNFQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.390716079 (291h23m17.94218653s) [noise=true , address=145.79.227.31:4799 , piece_id=CHOH62M4JKZGHNHR76M2WUKPE7G772IHGQG7UHYU5PVAYEKC4VPQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.403575166 (291h23m17.955045627s) [noise=true , address=145.79.227.10:4402 , piece_id=FDDET6WXFVXZJIG2YENY3RVK3SC2LEOSRCR25VLYINEWW5YFJHYQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.433618534 (291h23m17.985088985s) [noise=true , address=145.79.227.28:4726 , piece_id=72XULSSYQMZQAXFCRQFMXOBRCOE2LILOV6FPDORRPMWI6GVBVGHQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.756646770 (291h23m18.308117231s) [noise=true , address=145.79.227.16:4909 , piece_id=5JKO5HSDEUTJN4TK6CUZ273EYNR6A6Y3625TUUOS3VKHMAPWDHFQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.953171670 (291h23m18.504642121s) [noise=true , address=77.90.31.12:4589 , piece_id=S4JENPNCKPBMMRSMBD7AFKWWT6VI2DWM4YWE4O7IPTIZYEBZFOMA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.958723291 (291h23m18.510193762s) [noise=true , address=77.90.31.18:4805 , piece_id=3E62LR5OVVTS3O7CFDRDABMILFBBVE6PS74CXRMTPGC4CVAXNW2A, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.964914050 (291h23m18.516384511s) [noise=true , address=45.91.116.4:4940 , piece_id=OXPGVCFNLUECUPAETAAO6N42JULU6PJQJTXZU4PP6XFA3ENGUFYA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.981247682 (291h23m18.532718144s) [noise=true , address=45.91.116.12:4225 , piece_id=3OUXQAWMEGWO26XBUBRNAEHXV2DP5EUP5UUKNVRZHHPMASJ57OXQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.981267450 (291h23m18.532737911s) [seg_pos={0 0}, upload=256610]: Segment reached optimal threshold of 13 pieces. Nov 21 11:27:19.981527044 (291h23m18.532997505s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection Nov 21 11:27:19.981535420 (291h23m18.533005871s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981690315 (291h23m18.533160786s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Nov 21 11:27:19.981717316 (291h23m18.533187767s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981727796 (291h23m18.533198257s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 21 11:27:19.981735151 (291h23m18.533205602s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981884485 (291h23m18.533354946s) [seg_pos={0 0}, upload=256610]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:27:19.981964216 (291h23m18.533434677s) [upload=256610]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:27:19.981968945 (291h23m18.533439406s) [upload=256610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:27:20.271805188 (291h23m18.82327611s) [upload=256610]: Done uploading segments... Nov 21 11:27:20.271830285 (291h23m18.823300746s) [upload=256610]: Upload finished. err: <nil> Duration: 6.448769979s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 381.391532ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:31:19 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 381.42235ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 461.303458ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:31:19 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 461.333415ms edge gateway vhost bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 687.944594ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:31:12 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 687.974901ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 385.25629ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:30:55 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 385.283421ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [..................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000386589s trace: https://jaeger.dev.storj.io/trace/5738976329747081 FAIL (2025-11-21 11:30:46 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.000413529s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 475.553184ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:30:41 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 475.584101ms edge gateway vhost bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........................X...X.X......X.XX...X.X.X....X.....] duration: 8.049631391s trace: https://jaeger.dev.storj.io/trace/7e7a6aa5cbba947b FAIL (2025-11-21 11:30:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ov 21 11:27:17.125486410 (291h23m15.676956861s) [upload=256610]: Starting upload Nov 21 11:27:17.125579928 (291h23m15.677050399s) [upload=256610]: Done starting upload Nov 21 11:27:17.125690539 (291h23m15.677161s) [upload=256610]: Uploading segments... Nov 21 11:27:17.126036017 (291h23m15.677506468s) [upload=256610]: Got next segment. Inline: false Nov 21 11:27:17.126069811 (291h23m15.677540262s) [upload=256610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:27:17.126074891 (291h23m15.677545352s) [upload=256610]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:27:17.373868957 (291h23m15.925339418s) [seg_pos={0 0}, upload=256610]: Begin upload segment... Nov 21 11:27:17.373965912 (291h23m15.925436373s) [seg_pos={0 0}, upload=256610]: Done begin upload segment. Nov 21 11:27:17.373988064 (291h23m15.925458525s) [noise=true , address=77.90.31.18:4805 , piece_id=3E62LR5OVVTS3O7CFDRDABMILFBBVE6PS74CXRMTPGC4CVAXNW2A, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.374015958 (291h23m15.925486419s) [noise=true , address=145.79.227.23:4545 , piece_id=6MBV4336KPSHR3S67T333ZQCLXX5QBXOYAEBAUG76CVDU2CVDJXA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.374732552 (291h23m15.926203013s) [noise=true , address=45.91.116.12:4225 , piece_id=3OUXQAWMEGWO26XBUBRNAEHXV2DP5EUP5UUKNVRZHHPMASJ57OXQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375180444 (291h23m15.926650906s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375516435 (291h23m15.926986886s) [noise=true , address=145.79.227.24:4581 , piece_id=VVU2M3HLNS4A46SSIQRCOR6WV6J5FUHREINR3TC2H4RJBV4GZNFQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375888683 (291h23m15.927359144s) [noise=true , address=145.79.227.16:4909 , piece_id=5JKO5HSDEUTJN4TK6CUZ273EYNR6A6Y3625TUUOS3VKHMAPWDHFQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.376543971 (291h23m15.928014432s) [noise=true , address=145.79.227.21:4473 , piece_id=ILRSOD2EDSJXRMVZ7NITZOAZTRQQ5UUW3GKB3QB7DQASVRQVC27A, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.376981635 (291h23m15.928452096s) [noise=true , address=145.79.227.11:4731 , piece_id=CPBGYXE7CIJRVCLWCCMTL22SL2ZSC62KRQINNNACO4MYI4YLRGGQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.377562520 (291h23m15.929032981s) [noise=true , address=145.79.227.31:4799 , piece_id=CHOH62M4JKZGHNHR76M2WUKPE7G772IHGQG7UHYU5PVAYEKC4VPQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.378610416 (291h23m15.930080877s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWQ2CNVJCR5FKJ3OA5MKPH42O27BKLSTIJJIPKEQSDM3EJHNGRQA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.378974038 (291h23m15.930444499s) [noise=true , address=145.79.227.28:4726 , piece_id=72XULSSYQMZQAXFCRQFMXOBRCOE2LILOV6FPDORRPMWI6GVBVGHQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.379921312 (291h23m15.931391773s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.380443847 (291h23m15.931914308s) [noise=true , address=77.90.31.12:4589 , piece_id=S4JENPNCKPBMMRSMBD7AFKWWT6VI2DWM4YWE4O7IPTIZYEBZFOMA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.381260582 (291h23m15.932731044s) [noise=true , address=145.79.227.10:4402 , piece_id=FDDET6WXFVXZJIG2YENY3RVK3SC2LEOSRCR25VLYINEWW5YFJHYQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.381929115 (291h23m15.933399576s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.383439992 (291h23m15.934910473s) [noise=true , address=45.91.116.4:4940 , piece_id=OXPGVCFNLUECUPAETAAO6N42JULU6PJQJTXZU4PP6XFA3ENGUFYA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:18.997259485 (291h23m17.548729936s) [upload=256610]: Next returned nil segment Nov 21 11:27:18.997276468 (291h23m17.548746929s) [upload=256610]: Waiting for error group managing segments... Nov 21 11:27:19.376515178 (291h23m17.927985629s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWQ2CNVJCR5FKJ3OA5MKPH42O27BKLSTIJJIPKEQSDM3EJHNGRQA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.376615770 (291h23m17.928086231s) [noise=true , address=145.79.227.21:4473 , piece_id=ILRSOD2EDSJXRMVZ7NITZOAZTRQQ5UUW3GKB3QB7DQASVRQVC27A, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.380558629 (291h23m17.93202909s) [noise=true , address=145.79.227.11:4731 , piece_id=CPBGYXE7CIJRVCLWCCMTL22SL2ZSC62KRQINNNACO4MYI4YLRGGQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.386618928 (291h23m17.93808942s) [noise=true , address=145.79.227.23:4545 , piece_id=6MBV4336KPSHR3S67T333ZQCLXX5QBXOYAEBAUG76CVDU2CVDJXA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.388538574 (291h23m17.940009035s) [noise=true , address=145.79.227.24:4581 , piece_id=VVU2M3HLNS4A46SSIQRCOR6WV6J5FUHREINR3TC2H4RJBV4GZNFQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.390716079 (291h23m17.94218653s) [noise=true , address=145.79.227.31:4799 , piece_id=CHOH62M4JKZGHNHR76M2WUKPE7G772IHGQG7UHYU5PVAYEKC4VPQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.403575166 (291h23m17.955045627s) [noise=true , address=145.79.227.10:4402 , piece_id=FDDET6WXFVXZJIG2YENY3RVK3SC2LEOSRCR25VLYINEWW5YFJHYQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.433618534 (291h23m17.985088985s) [noise=true , address=145.79.227.28:4726 , piece_id=72XULSSYQMZQAXFCRQFMXOBRCOE2LILOV6FPDORRPMWI6GVBVGHQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.756646770 (291h23m18.308117231s) [noise=true , address=145.79.227.16:4909 , piece_id=5JKO5HSDEUTJN4TK6CUZ273EYNR6A6Y3625TUUOS3VKHMAPWDHFQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.953171670 (291h23m18.504642121s) [noise=true , address=77.90.31.12:4589 , piece_id=S4JENPNCKPBMMRSMBD7AFKWWT6VI2DWM4YWE4O7IPTIZYEBZFOMA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.958723291 (291h23m18.510193762s) [noise=true , address=77.90.31.18:4805 , piece_id=3E62LR5OVVTS3O7CFDRDABMILFBBVE6PS74CXRMTPGC4CVAXNW2A, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.964914050 (291h23m18.516384511s) [noise=true , address=45.91.116.4:4940 , piece_id=OXPGVCFNLUECUPAETAAO6N42JULU6PJQJTXZU4PP6XFA3ENGUFYA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.981247682 (291h23m18.532718144s) [noise=true , address=45.91.116.12:4225 , piece_id=3OUXQAWMEGWO26XBUBRNAEHXV2DP5EUP5UUKNVRZHHPMASJ57OXQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.981267450 (291h23m18.532737911s) [seg_pos={0 0}, upload=256610]: Segment reached optimal threshold of 13 pieces. Nov 21 11:27:19.981527044 (291h23m18.532997505s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection Nov 21 11:27:19.981535420 (291h23m18.533005871s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981690315 (291h23m18.533160786s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Nov 21 11:27:19.981717316 (291h23m18.533187767s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981727796 (291h23m18.533198257s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 21 11:27:19.981735151 (291h23m18.533205602s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981884485 (291h23m18.533354946s) [seg_pos={0 0}, upload=256610]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:27:19.981964216 (291h23m18.533434677s) [upload=256610]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:27:19.981968945 (291h23m18.533439406s) [upload=256610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:27:20.271805188 (291h23m18.82327611s) [upload=256610]: Done uploading segments... Nov 21 11:27:20.271830285 (291h23m18.823300746s) [upload=256610]: Upload finished. err: <nil> Duration: 8.049670145s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001363211s trace: https://jaeger.dev.storj.io/trace/32f9f56a8cdf987a FAIL (2025-11-21 11:30:23 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.001398858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 4.99256495s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:30:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 4.992595618s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 389.310039ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:30:15 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 389.380312ms edge linksharing bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 491.068596ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:30:12 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 491.090117ms edge gateway vhost bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001190179s trace: https://jaeger.dev.storj.io/trace/cc9fc12c6bd58cd FAIL (2025-11-21 11:30:06 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.001219235s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 478.713288ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:30:05 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 478.744538ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........................X...X.X......X.XX...X.X.X....X....] duration: 6.294632341s trace: https://jaeger.dev.storj.io/trace/7e7a6aa5cbba947b FAIL (2025-11-21 11:29:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ov 21 11:27:17.125486410 (291h23m15.676956861s) [upload=256610]: Starting upload Nov 21 11:27:17.125579928 (291h23m15.677050399s) [upload=256610]: Done starting upload Nov 21 11:27:17.125690539 (291h23m15.677161s) [upload=256610]: Uploading segments... Nov 21 11:27:17.126036017 (291h23m15.677506468s) [upload=256610]: Got next segment. Inline: false Nov 21 11:27:17.126069811 (291h23m15.677540262s) [upload=256610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:27:17.126074891 (291h23m15.677545352s) [upload=256610]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:27:17.373868957 (291h23m15.925339418s) [seg_pos={0 0}, upload=256610]: Begin upload segment... Nov 21 11:27:17.373965912 (291h23m15.925436373s) [seg_pos={0 0}, upload=256610]: Done begin upload segment. Nov 21 11:27:17.373988064 (291h23m15.925458525s) [noise=true , address=77.90.31.18:4805 , piece_id=3E62LR5OVVTS3O7CFDRDABMILFBBVE6PS74CXRMTPGC4CVAXNW2A, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.374015958 (291h23m15.925486419s) [noise=true , address=145.79.227.23:4545 , piece_id=6MBV4336KPSHR3S67T333ZQCLXX5QBXOYAEBAUG76CVDU2CVDJXA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.374732552 (291h23m15.926203013s) [noise=true , address=45.91.116.12:4225 , piece_id=3OUXQAWMEGWO26XBUBRNAEHXV2DP5EUP5UUKNVRZHHPMASJ57OXQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375180444 (291h23m15.926650906s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375516435 (291h23m15.926986886s) [noise=true , address=145.79.227.24:4581 , piece_id=VVU2M3HLNS4A46SSIQRCOR6WV6J5FUHREINR3TC2H4RJBV4GZNFQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375888683 (291h23m15.927359144s) [noise=true , address=145.79.227.16:4909 , piece_id=5JKO5HSDEUTJN4TK6CUZ273EYNR6A6Y3625TUUOS3VKHMAPWDHFQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.376543971 (291h23m15.928014432s) [noise=true , address=145.79.227.21:4473 , piece_id=ILRSOD2EDSJXRMVZ7NITZOAZTRQQ5UUW3GKB3QB7DQASVRQVC27A, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.376981635 (291h23m15.928452096s) [noise=true , address=145.79.227.11:4731 , piece_id=CPBGYXE7CIJRVCLWCCMTL22SL2ZSC62KRQINNNACO4MYI4YLRGGQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.377562520 (291h23m15.929032981s) [noise=true , address=145.79.227.31:4799 , piece_id=CHOH62M4JKZGHNHR76M2WUKPE7G772IHGQG7UHYU5PVAYEKC4VPQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.378610416 (291h23m15.930080877s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWQ2CNVJCR5FKJ3OA5MKPH42O27BKLSTIJJIPKEQSDM3EJHNGRQA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.378974038 (291h23m15.930444499s) [noise=true , address=145.79.227.28:4726 , piece_id=72XULSSYQMZQAXFCRQFMXOBRCOE2LILOV6FPDORRPMWI6GVBVGHQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.379921312 (291h23m15.931391773s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.380443847 (291h23m15.931914308s) [noise=true , address=77.90.31.12:4589 , piece_id=S4JENPNCKPBMMRSMBD7AFKWWT6VI2DWM4YWE4O7IPTIZYEBZFOMA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.381260582 (291h23m15.932731044s) [noise=true , address=145.79.227.10:4402 , piece_id=FDDET6WXFVXZJIG2YENY3RVK3SC2LEOSRCR25VLYINEWW5YFJHYQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.381929115 (291h23m15.933399576s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.383439992 (291h23m15.934910473s) [noise=true , address=45.91.116.4:4940 , piece_id=OXPGVCFNLUECUPAETAAO6N42JULU6PJQJTXZU4PP6XFA3ENGUFYA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:18.997259485 (291h23m17.548729936s) [upload=256610]: Next returned nil segment Nov 21 11:27:18.997276468 (291h23m17.548746929s) [upload=256610]: Waiting for error group managing segments... Nov 21 11:27:19.376515178 (291h23m17.927985629s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWQ2CNVJCR5FKJ3OA5MKPH42O27BKLSTIJJIPKEQSDM3EJHNGRQA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.376615770 (291h23m17.928086231s) [noise=true , address=145.79.227.21:4473 , piece_id=ILRSOD2EDSJXRMVZ7NITZOAZTRQQ5UUW3GKB3QB7DQASVRQVC27A, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.380558629 (291h23m17.93202909s) [noise=true , address=145.79.227.11:4731 , piece_id=CPBGYXE7CIJRVCLWCCMTL22SL2ZSC62KRQINNNACO4MYI4YLRGGQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.386618928 (291h23m17.93808942s) [noise=true , address=145.79.227.23:4545 , piece_id=6MBV4336KPSHR3S67T333ZQCLXX5QBXOYAEBAUG76CVDU2CVDJXA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.388538574 (291h23m17.940009035s) [noise=true , address=145.79.227.24:4581 , piece_id=VVU2M3HLNS4A46SSIQRCOR6WV6J5FUHREINR3TC2H4RJBV4GZNFQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.390716079 (291h23m17.94218653s) [noise=true , address=145.79.227.31:4799 , piece_id=CHOH62M4JKZGHNHR76M2WUKPE7G772IHGQG7UHYU5PVAYEKC4VPQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.403575166 (291h23m17.955045627s) [noise=true , address=145.79.227.10:4402 , piece_id=FDDET6WXFVXZJIG2YENY3RVK3SC2LEOSRCR25VLYINEWW5YFJHYQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.433618534 (291h23m17.985088985s) [noise=true , address=145.79.227.28:4726 , piece_id=72XULSSYQMZQAXFCRQFMXOBRCOE2LILOV6FPDORRPMWI6GVBVGHQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.756646770 (291h23m18.308117231s) [noise=true , address=145.79.227.16:4909 , piece_id=5JKO5HSDEUTJN4TK6CUZ273EYNR6A6Y3625TUUOS3VKHMAPWDHFQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.953171670 (291h23m18.504642121s) [noise=true , address=77.90.31.12:4589 , piece_id=S4JENPNCKPBMMRSMBD7AFKWWT6VI2DWM4YWE4O7IPTIZYEBZFOMA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.958723291 (291h23m18.510193762s) [noise=true , address=77.90.31.18:4805 , piece_id=3E62LR5OVVTS3O7CFDRDABMILFBBVE6PS74CXRMTPGC4CVAXNW2A, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.964914050 (291h23m18.516384511s) [noise=true , address=45.91.116.4:4940 , piece_id=OXPGVCFNLUECUPAETAAO6N42JULU6PJQJTXZU4PP6XFA3ENGUFYA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.981247682 (291h23m18.532718144s) [noise=true , address=45.91.116.12:4225 , piece_id=3OUXQAWMEGWO26XBUBRNAEHXV2DP5EUP5UUKNVRZHHPMASJ57OXQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.981267450 (291h23m18.532737911s) [seg_pos={0 0}, upload=256610]: Segment reached optimal threshold of 13 pieces. Nov 21 11:27:19.981527044 (291h23m18.532997505s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection Nov 21 11:27:19.981535420 (291h23m18.533005871s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981690315 (291h23m18.533160786s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Nov 21 11:27:19.981717316 (291h23m18.533187767s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981727796 (291h23m18.533198257s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 21 11:27:19.981735151 (291h23m18.533205602s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981884485 (291h23m18.533354946s) [seg_pos={0 0}, upload=256610]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:27:19.981964216 (291h23m18.533434677s) [upload=256610]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:27:19.981968945 (291h23m18.533439406s) [upload=256610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:27:20.271805188 (291h23m18.82327611s) [upload=256610]: Done uploading segments... Nov 21 11:27:20.271830285 (291h23m18.823300746s) [upload=256610]: Upload finished. err: <nil> Duration: 6.294664633s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 381.658098ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:29:48 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 381.691512ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 387.325811ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:29:46 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 387.343304ms edge linksharing bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 517.062905ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:29:43 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 517.114914ms edge gateway vhost bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 470.12807ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:29:42 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 470.155783ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 5.877452786s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:29:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.877487964s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001342529s trace: https://jaeger.dev.storj.io/trace/4bbe3db20da11e50 FAIL (2025-11-21 11:29:19 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.001370401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........................X...X.X......X.XX...X.X.X....X...] duration: 6.31738561s trace: https://jaeger.dev.storj.io/trace/7e7a6aa5cbba947b FAIL (2025-11-21 11:29:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ov 21 11:27:17.125486410 (291h23m15.676956861s) [upload=256610]: Starting upload Nov 21 11:27:17.125579928 (291h23m15.677050399s) [upload=256610]: Done starting upload Nov 21 11:27:17.125690539 (291h23m15.677161s) [upload=256610]: Uploading segments... Nov 21 11:27:17.126036017 (291h23m15.677506468s) [upload=256610]: Got next segment. Inline: false Nov 21 11:27:17.126069811 (291h23m15.677540262s) [upload=256610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:27:17.126074891 (291h23m15.677545352s) [upload=256610]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:27:17.373868957 (291h23m15.925339418s) [seg_pos={0 0}, upload=256610]: Begin upload segment... Nov 21 11:27:17.373965912 (291h23m15.925436373s) [seg_pos={0 0}, upload=256610]: Done begin upload segment. Nov 21 11:27:17.373988064 (291h23m15.925458525s) [noise=true , address=77.90.31.18:4805 , piece_id=3E62LR5OVVTS3O7CFDRDABMILFBBVE6PS74CXRMTPGC4CVAXNW2A, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.374015958 (291h23m15.925486419s) [noise=true , address=145.79.227.23:4545 , piece_id=6MBV4336KPSHR3S67T333ZQCLXX5QBXOYAEBAUG76CVDU2CVDJXA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.374732552 (291h23m15.926203013s) [noise=true , address=45.91.116.12:4225 , piece_id=3OUXQAWMEGWO26XBUBRNAEHXV2DP5EUP5UUKNVRZHHPMASJ57OXQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375180444 (291h23m15.926650906s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375516435 (291h23m15.926986886s) [noise=true , address=145.79.227.24:4581 , piece_id=VVU2M3HLNS4A46SSIQRCOR6WV6J5FUHREINR3TC2H4RJBV4GZNFQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375888683 (291h23m15.927359144s) [noise=true , address=145.79.227.16:4909 , piece_id=5JKO5HSDEUTJN4TK6CUZ273EYNR6A6Y3625TUUOS3VKHMAPWDHFQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.376543971 (291h23m15.928014432s) [noise=true , address=145.79.227.21:4473 , piece_id=ILRSOD2EDSJXRMVZ7NITZOAZTRQQ5UUW3GKB3QB7DQASVRQVC27A, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.376981635 (291h23m15.928452096s) [noise=true , address=145.79.227.11:4731 , piece_id=CPBGYXE7CIJRVCLWCCMTL22SL2ZSC62KRQINNNACO4MYI4YLRGGQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.377562520 (291h23m15.929032981s) [noise=true , address=145.79.227.31:4799 , piece_id=CHOH62M4JKZGHNHR76M2WUKPE7G772IHGQG7UHYU5PVAYEKC4VPQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.378610416 (291h23m15.930080877s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWQ2CNVJCR5FKJ3OA5MKPH42O27BKLSTIJJIPKEQSDM3EJHNGRQA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.378974038 (291h23m15.930444499s) [noise=true , address=145.79.227.28:4726 , piece_id=72XULSSYQMZQAXFCRQFMXOBRCOE2LILOV6FPDORRPMWI6GVBVGHQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.379921312 (291h23m15.931391773s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.380443847 (291h23m15.931914308s) [noise=true , address=77.90.31.12:4589 , piece_id=S4JENPNCKPBMMRSMBD7AFKWWT6VI2DWM4YWE4O7IPTIZYEBZFOMA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.381260582 (291h23m15.932731044s) [noise=true , address=145.79.227.10:4402 , piece_id=FDDET6WXFVXZJIG2YENY3RVK3SC2LEOSRCR25VLYINEWW5YFJHYQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.381929115 (291h23m15.933399576s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.383439992 (291h23m15.934910473s) [noise=true , address=45.91.116.4:4940 , piece_id=OXPGVCFNLUECUPAETAAO6N42JULU6PJQJTXZU4PP6XFA3ENGUFYA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:18.997259485 (291h23m17.548729936s) [upload=256610]: Next returned nil segment Nov 21 11:27:18.997276468 (291h23m17.548746929s) [upload=256610]: Waiting for error group managing segments... Nov 21 11:27:19.376515178 (291h23m17.927985629s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWQ2CNVJCR5FKJ3OA5MKPH42O27BKLSTIJJIPKEQSDM3EJHNGRQA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.376615770 (291h23m17.928086231s) [noise=true , address=145.79.227.21:4473 , piece_id=ILRSOD2EDSJXRMVZ7NITZOAZTRQQ5UUW3GKB3QB7DQASVRQVC27A, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.380558629 (291h23m17.93202909s) [noise=true , address=145.79.227.11:4731 , piece_id=CPBGYXE7CIJRVCLWCCMTL22SL2ZSC62KRQINNNACO4MYI4YLRGGQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.386618928 (291h23m17.93808942s) [noise=true , address=145.79.227.23:4545 , piece_id=6MBV4336KPSHR3S67T333ZQCLXX5QBXOYAEBAUG76CVDU2CVDJXA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.388538574 (291h23m17.940009035s) [noise=true , address=145.79.227.24:4581 , piece_id=VVU2M3HLNS4A46SSIQRCOR6WV6J5FUHREINR3TC2H4RJBV4GZNFQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.390716079 (291h23m17.94218653s) [noise=true , address=145.79.227.31:4799 , piece_id=CHOH62M4JKZGHNHR76M2WUKPE7G772IHGQG7UHYU5PVAYEKC4VPQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.403575166 (291h23m17.955045627s) [noise=true , address=145.79.227.10:4402 , piece_id=FDDET6WXFVXZJIG2YENY3RVK3SC2LEOSRCR25VLYINEWW5YFJHYQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.433618534 (291h23m17.985088985s) [noise=true , address=145.79.227.28:4726 , piece_id=72XULSSYQMZQAXFCRQFMXOBRCOE2LILOV6FPDORRPMWI6GVBVGHQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.756646770 (291h23m18.308117231s) [noise=true , address=145.79.227.16:4909 , piece_id=5JKO5HSDEUTJN4TK6CUZ273EYNR6A6Y3625TUUOS3VKHMAPWDHFQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.953171670 (291h23m18.504642121s) [noise=true , address=77.90.31.12:4589 , piece_id=S4JENPNCKPBMMRSMBD7AFKWWT6VI2DWM4YWE4O7IPTIZYEBZFOMA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.958723291 (291h23m18.510193762s) [noise=true , address=77.90.31.18:4805 , piece_id=3E62LR5OVVTS3O7CFDRDABMILFBBVE6PS74CXRMTPGC4CVAXNW2A, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.964914050 (291h23m18.516384511s) [noise=true , address=45.91.116.4:4940 , piece_id=OXPGVCFNLUECUPAETAAO6N42JULU6PJQJTXZU4PP6XFA3ENGUFYA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.981247682 (291h23m18.532718144s) [noise=true , address=45.91.116.12:4225 , piece_id=3OUXQAWMEGWO26XBUBRNAEHXV2DP5EUP5UUKNVRZHHPMASJ57OXQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.981267450 (291h23m18.532737911s) [seg_pos={0 0}, upload=256610]: Segment reached optimal threshold of 13 pieces. Nov 21 11:27:19.981527044 (291h23m18.532997505s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection Nov 21 11:27:19.981535420 (291h23m18.533005871s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981690315 (291h23m18.533160786s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Nov 21 11:27:19.981717316 (291h23m18.533187767s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981727796 (291h23m18.533198257s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 21 11:27:19.981735151 (291h23m18.533205602s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981884485 (291h23m18.533354946s) [seg_pos={0 0}, upload=256610]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:27:19.981964216 (291h23m18.533434677s) [upload=256610]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:27:19.981968945 (291h23m18.533439406s) [upload=256610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:27:20.271805188 (291h23m18.82327611s) [upload=256610]: Done uploading segments... Nov 21 11:27:20.271830285 (291h23m18.823300746s) [upload=256610]: Upload finished. err: <nil> Duration: 6.317425174s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 401.125508ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:29:01 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 401.157368ms edge linksharing bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 463.744787ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:28:56 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 463.845418ms edge gateway vhost bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [............................X...X.X......X.XX...X.X.X....X..] duration: 6.257612852s trace: https://jaeger.dev.storj.io/trace/7e7a6aa5cbba947b FAIL (2025-11-21 11:28:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ov 21 11:27:17.125486410 (291h23m15.676956861s) [upload=256610]: Starting upload Nov 21 11:27:17.125579928 (291h23m15.677050399s) [upload=256610]: Done starting upload Nov 21 11:27:17.125690539 (291h23m15.677161s) [upload=256610]: Uploading segments... Nov 21 11:27:17.126036017 (291h23m15.677506468s) [upload=256610]: Got next segment. Inline: false Nov 21 11:27:17.126069811 (291h23m15.677540262s) [upload=256610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:27:17.126074891 (291h23m15.677545352s) [upload=256610]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:27:17.373868957 (291h23m15.925339418s) [seg_pos={0 0}, upload=256610]: Begin upload segment... Nov 21 11:27:17.373965912 (291h23m15.925436373s) [seg_pos={0 0}, upload=256610]: Done begin upload segment. Nov 21 11:27:17.373988064 (291h23m15.925458525s) [noise=true , address=77.90.31.18:4805 , piece_id=3E62LR5OVVTS3O7CFDRDABMILFBBVE6PS74CXRMTPGC4CVAXNW2A, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.374015958 (291h23m15.925486419s) [noise=true , address=145.79.227.23:4545 , piece_id=6MBV4336KPSHR3S67T333ZQCLXX5QBXOYAEBAUG76CVDU2CVDJXA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.374732552 (291h23m15.926203013s) [noise=true , address=45.91.116.12:4225 , piece_id=3OUXQAWMEGWO26XBUBRNAEHXV2DP5EUP5UUKNVRZHHPMASJ57OXQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375180444 (291h23m15.926650906s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375516435 (291h23m15.926986886s) [noise=true , address=145.79.227.24:4581 , piece_id=VVU2M3HLNS4A46SSIQRCOR6WV6J5FUHREINR3TC2H4RJBV4GZNFQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375888683 (291h23m15.927359144s) [noise=true , address=145.79.227.16:4909 , piece_id=5JKO5HSDEUTJN4TK6CUZ273EYNR6A6Y3625TUUOS3VKHMAPWDHFQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.376543971 (291h23m15.928014432s) [noise=true , address=145.79.227.21:4473 , piece_id=ILRSOD2EDSJXRMVZ7NITZOAZTRQQ5UUW3GKB3QB7DQASVRQVC27A, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.376981635 (291h23m15.928452096s) [noise=true , address=145.79.227.11:4731 , piece_id=CPBGYXE7CIJRVCLWCCMTL22SL2ZSC62KRQINNNACO4MYI4YLRGGQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.377562520 (291h23m15.929032981s) [noise=true , address=145.79.227.31:4799 , piece_id=CHOH62M4JKZGHNHR76M2WUKPE7G772IHGQG7UHYU5PVAYEKC4VPQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.378610416 (291h23m15.930080877s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWQ2CNVJCR5FKJ3OA5MKPH42O27BKLSTIJJIPKEQSDM3EJHNGRQA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.378974038 (291h23m15.930444499s) [noise=true , address=145.79.227.28:4726 , piece_id=72XULSSYQMZQAXFCRQFMXOBRCOE2LILOV6FPDORRPMWI6GVBVGHQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.379921312 (291h23m15.931391773s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.380443847 (291h23m15.931914308s) [noise=true , address=77.90.31.12:4589 , piece_id=S4JENPNCKPBMMRSMBD7AFKWWT6VI2DWM4YWE4O7IPTIZYEBZFOMA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.381260582 (291h23m15.932731044s) [noise=true , address=145.79.227.10:4402 , piece_id=FDDET6WXFVXZJIG2YENY3RVK3SC2LEOSRCR25VLYINEWW5YFJHYQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.381929115 (291h23m15.933399576s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.383439992 (291h23m15.934910473s) [noise=true , address=45.91.116.4:4940 , piece_id=OXPGVCFNLUECUPAETAAO6N42JULU6PJQJTXZU4PP6XFA3ENGUFYA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:18.997259485 (291h23m17.548729936s) [upload=256610]: Next returned nil segment Nov 21 11:27:18.997276468 (291h23m17.548746929s) [upload=256610]: Waiting for error group managing segments... Nov 21 11:27:19.376515178 (291h23m17.927985629s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWQ2CNVJCR5FKJ3OA5MKPH42O27BKLSTIJJIPKEQSDM3EJHNGRQA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.376615770 (291h23m17.928086231s) [noise=true , address=145.79.227.21:4473 , piece_id=ILRSOD2EDSJXRMVZ7NITZOAZTRQQ5UUW3GKB3QB7DQASVRQVC27A, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.380558629 (291h23m17.93202909s) [noise=true , address=145.79.227.11:4731 , piece_id=CPBGYXE7CIJRVCLWCCMTL22SL2ZSC62KRQINNNACO4MYI4YLRGGQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.386618928 (291h23m17.93808942s) [noise=true , address=145.79.227.23:4545 , piece_id=6MBV4336KPSHR3S67T333ZQCLXX5QBXOYAEBAUG76CVDU2CVDJXA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.388538574 (291h23m17.940009035s) [noise=true , address=145.79.227.24:4581 , piece_id=VVU2M3HLNS4A46SSIQRCOR6WV6J5FUHREINR3TC2H4RJBV4GZNFQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.390716079 (291h23m17.94218653s) [noise=true , address=145.79.227.31:4799 , piece_id=CHOH62M4JKZGHNHR76M2WUKPE7G772IHGQG7UHYU5PVAYEKC4VPQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.403575166 (291h23m17.955045627s) [noise=true , address=145.79.227.10:4402 , piece_id=FDDET6WXFVXZJIG2YENY3RVK3SC2LEOSRCR25VLYINEWW5YFJHYQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.433618534 (291h23m17.985088985s) [noise=true , address=145.79.227.28:4726 , piece_id=72XULSSYQMZQAXFCRQFMXOBRCOE2LILOV6FPDORRPMWI6GVBVGHQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.756646770 (291h23m18.308117231s) [noise=true , address=145.79.227.16:4909 , piece_id=5JKO5HSDEUTJN4TK6CUZ273EYNR6A6Y3625TUUOS3VKHMAPWDHFQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.953171670 (291h23m18.504642121s) [noise=true , address=77.90.31.12:4589 , piece_id=S4JENPNCKPBMMRSMBD7AFKWWT6VI2DWM4YWE4O7IPTIZYEBZFOMA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.958723291 (291h23m18.510193762s) [noise=true , address=77.90.31.18:4805 , piece_id=3E62LR5OVVTS3O7CFDRDABMILFBBVE6PS74CXRMTPGC4CVAXNW2A, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.964914050 (291h23m18.516384511s) [noise=true , address=45.91.116.4:4940 , piece_id=OXPGVCFNLUECUPAETAAO6N42JULU6PJQJTXZU4PP6XFA3ENGUFYA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.981247682 (291h23m18.532718144s) [noise=true , address=45.91.116.12:4225 , piece_id=3OUXQAWMEGWO26XBUBRNAEHXV2DP5EUP5UUKNVRZHHPMASJ57OXQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.981267450 (291h23m18.532737911s) [seg_pos={0 0}, upload=256610]: Segment reached optimal threshold of 13 pieces. Nov 21 11:27:19.981527044 (291h23m18.532997505s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection Nov 21 11:27:19.981535420 (291h23m18.533005871s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981690315 (291h23m18.533160786s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Nov 21 11:27:19.981717316 (291h23m18.533187767s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981727796 (291h23m18.533198257s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 21 11:27:19.981735151 (291h23m18.533205602s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981884485 (291h23m18.533354946s) [seg_pos={0 0}, upload=256610]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:27:19.981964216 (291h23m18.533434677s) [upload=256610]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:27:19.981968945 (291h23m18.533439406s) [upload=256610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:27:20.271805188 (291h23m18.82327611s) [upload=256610]: Done uploading segments... Nov 21 11:27:20.271830285 (291h23m18.823300746s) [upload=256610]: Upload finished. err: <nil> Duration: 6.257723222s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 5.714921204s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:28:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.71495063s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 474.537879ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:28:41 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 474.56476ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 390.717738ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:28:35 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 390.747666ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001280592s trace: https://jaeger.dev.storj.io/trace/16d017a648a9fd36 FAIL (2025-11-21 11:28:25 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.001311391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 5.855096017s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:28:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.855125984s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 464.547665ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:27:59 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 464.575919ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.............................X...X.X......X.XX...X.X.X....X.] duration: 6.584840236s trace: https://jaeger.dev.storj.io/trace/7e7a6aa5cbba947b FAIL (2025-11-21 11:27:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ov 21 11:27:17.125486410 (291h23m15.676956861s) [upload=256610]: Starting upload Nov 21 11:27:17.125579928 (291h23m15.677050399s) [upload=256610]: Done starting upload Nov 21 11:27:17.125690539 (291h23m15.677161s) [upload=256610]: Uploading segments... Nov 21 11:27:17.126036017 (291h23m15.677506468s) [upload=256610]: Got next segment. Inline: false Nov 21 11:27:17.126069811 (291h23m15.677540262s) [upload=256610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:27:17.126074891 (291h23m15.677545352s) [upload=256610]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:27:17.373868957 (291h23m15.925339418s) [seg_pos={0 0}, upload=256610]: Begin upload segment... Nov 21 11:27:17.373965912 (291h23m15.925436373s) [seg_pos={0 0}, upload=256610]: Done begin upload segment. Nov 21 11:27:17.373988064 (291h23m15.925458525s) [noise=true , address=77.90.31.18:4805 , piece_id=3E62LR5OVVTS3O7CFDRDABMILFBBVE6PS74CXRMTPGC4CVAXNW2A, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.374015958 (291h23m15.925486419s) [noise=true , address=145.79.227.23:4545 , piece_id=6MBV4336KPSHR3S67T333ZQCLXX5QBXOYAEBAUG76CVDU2CVDJXA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.374732552 (291h23m15.926203013s) [noise=true , address=45.91.116.12:4225 , piece_id=3OUXQAWMEGWO26XBUBRNAEHXV2DP5EUP5UUKNVRZHHPMASJ57OXQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375180444 (291h23m15.926650906s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375516435 (291h23m15.926986886s) [noise=true , address=145.79.227.24:4581 , piece_id=VVU2M3HLNS4A46SSIQRCOR6WV6J5FUHREINR3TC2H4RJBV4GZNFQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375888683 (291h23m15.927359144s) [noise=true , address=145.79.227.16:4909 , piece_id=5JKO5HSDEUTJN4TK6CUZ273EYNR6A6Y3625TUUOS3VKHMAPWDHFQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.376543971 (291h23m15.928014432s) [noise=true , address=145.79.227.21:4473 , piece_id=ILRSOD2EDSJXRMVZ7NITZOAZTRQQ5UUW3GKB3QB7DQASVRQVC27A, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.376981635 (291h23m15.928452096s) [noise=true , address=145.79.227.11:4731 , piece_id=CPBGYXE7CIJRVCLWCCMTL22SL2ZSC62KRQINNNACO4MYI4YLRGGQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.377562520 (291h23m15.929032981s) [noise=true , address=145.79.227.31:4799 , piece_id=CHOH62M4JKZGHNHR76M2WUKPE7G772IHGQG7UHYU5PVAYEKC4VPQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.378610416 (291h23m15.930080877s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWQ2CNVJCR5FKJ3OA5MKPH42O27BKLSTIJJIPKEQSDM3EJHNGRQA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.378974038 (291h23m15.930444499s) [noise=true , address=145.79.227.28:4726 , piece_id=72XULSSYQMZQAXFCRQFMXOBRCOE2LILOV6FPDORRPMWI6GVBVGHQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.379921312 (291h23m15.931391773s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.380443847 (291h23m15.931914308s) [noise=true , address=77.90.31.12:4589 , piece_id=S4JENPNCKPBMMRSMBD7AFKWWT6VI2DWM4YWE4O7IPTIZYEBZFOMA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.381260582 (291h23m15.932731044s) [noise=true , address=145.79.227.10:4402 , piece_id=FDDET6WXFVXZJIG2YENY3RVK3SC2LEOSRCR25VLYINEWW5YFJHYQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.381929115 (291h23m15.933399576s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.383439992 (291h23m15.934910473s) [noise=true , address=45.91.116.4:4940 , piece_id=OXPGVCFNLUECUPAETAAO6N42JULU6PJQJTXZU4PP6XFA3ENGUFYA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:18.997259485 (291h23m17.548729936s) [upload=256610]: Next returned nil segment Nov 21 11:27:18.997276468 (291h23m17.548746929s) [upload=256610]: Waiting for error group managing segments... Nov 21 11:27:19.376515178 (291h23m17.927985629s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWQ2CNVJCR5FKJ3OA5MKPH42O27BKLSTIJJIPKEQSDM3EJHNGRQA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.376615770 (291h23m17.928086231s) [noise=true , address=145.79.227.21:4473 , piece_id=ILRSOD2EDSJXRMVZ7NITZOAZTRQQ5UUW3GKB3QB7DQASVRQVC27A, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.380558629 (291h23m17.93202909s) [noise=true , address=145.79.227.11:4731 , piece_id=CPBGYXE7CIJRVCLWCCMTL22SL2ZSC62KRQINNNACO4MYI4YLRGGQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.386618928 (291h23m17.93808942s) [noise=true , address=145.79.227.23:4545 , piece_id=6MBV4336KPSHR3S67T333ZQCLXX5QBXOYAEBAUG76CVDU2CVDJXA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.388538574 (291h23m17.940009035s) [noise=true , address=145.79.227.24:4581 , piece_id=VVU2M3HLNS4A46SSIQRCOR6WV6J5FUHREINR3TC2H4RJBV4GZNFQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.390716079 (291h23m17.94218653s) [noise=true , address=145.79.227.31:4799 , piece_id=CHOH62M4JKZGHNHR76M2WUKPE7G772IHGQG7UHYU5PVAYEKC4VPQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.403575166 (291h23m17.955045627s) [noise=true , address=145.79.227.10:4402 , piece_id=FDDET6WXFVXZJIG2YENY3RVK3SC2LEOSRCR25VLYINEWW5YFJHYQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.433618534 (291h23m17.985088985s) [noise=true , address=145.79.227.28:4726 , piece_id=72XULSSYQMZQAXFCRQFMXOBRCOE2LILOV6FPDORRPMWI6GVBVGHQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.756646770 (291h23m18.308117231s) [noise=true , address=145.79.227.16:4909 , piece_id=5JKO5HSDEUTJN4TK6CUZ273EYNR6A6Y3625TUUOS3VKHMAPWDHFQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.953171670 (291h23m18.504642121s) [noise=true , address=77.90.31.12:4589 , piece_id=S4JENPNCKPBMMRSMBD7AFKWWT6VI2DWM4YWE4O7IPTIZYEBZFOMA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.958723291 (291h23m18.510193762s) [noise=true , address=77.90.31.18:4805 , piece_id=3E62LR5OVVTS3O7CFDRDABMILFBBVE6PS74CXRMTPGC4CVAXNW2A, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.964914050 (291h23m18.516384511s) [noise=true , address=45.91.116.4:4940 , piece_id=OXPGVCFNLUECUPAETAAO6N42JULU6PJQJTXZU4PP6XFA3ENGUFYA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.981247682 (291h23m18.532718144s) [noise=true , address=45.91.116.12:4225 , piece_id=3OUXQAWMEGWO26XBUBRNAEHXV2DP5EUP5UUKNVRZHHPMASJ57OXQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.981267450 (291h23m18.532737911s) [seg_pos={0 0}, upload=256610]: Segment reached optimal threshold of 13 pieces. Nov 21 11:27:19.981527044 (291h23m18.532997505s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection Nov 21 11:27:19.981535420 (291h23m18.533005871s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981690315 (291h23m18.533160786s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Nov 21 11:27:19.981717316 (291h23m18.533187767s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981727796 (291h23m18.533198257s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 21 11:27:19.981735151 (291h23m18.533205602s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981884485 (291h23m18.533354946s) [seg_pos={0 0}, upload=256610]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:27:19.981964216 (291h23m18.533434677s) [upload=256610]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:27:19.981968945 (291h23m18.533439406s) [upload=256610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:27:20.271805188 (291h23m18.82327611s) [upload=256610]: Done uploading segments... Nov 21 11:27:20.271830285 (291h23m18.823300746s) [upload=256610]: Upload finished. err: <nil> Duration: 6.584858721s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 436.924336ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:27:47 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 436.952329ms edge linksharing bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..............................X...X.X......X.XX...X.X.X....X] duration: 30.002914569s trace: https://jaeger.dev.storj.io/trace/7e7a6aa5cbba947b FAIL (2025-11-21 11:27:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ov 21 11:27:17.125486410 (291h23m15.676956861s) [upload=256610]: Starting upload Nov 21 11:27:17.125579928 (291h23m15.677050399s) [upload=256610]: Done starting upload Nov 21 11:27:17.125690539 (291h23m15.677161s) [upload=256610]: Uploading segments... Nov 21 11:27:17.126036017 (291h23m15.677506468s) [upload=256610]: Got next segment. Inline: false Nov 21 11:27:17.126069811 (291h23m15.677540262s) [upload=256610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:27:17.126074891 (291h23m15.677545352s) [upload=256610]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:27:17.373868957 (291h23m15.925339418s) [seg_pos={0 0}, upload=256610]: Begin upload segment... Nov 21 11:27:17.373965912 (291h23m15.925436373s) [seg_pos={0 0}, upload=256610]: Done begin upload segment. Nov 21 11:27:17.373988064 (291h23m15.925458525s) [noise=true , address=77.90.31.18:4805 , piece_id=3E62LR5OVVTS3O7CFDRDABMILFBBVE6PS74CXRMTPGC4CVAXNW2A, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.374015958 (291h23m15.925486419s) [noise=true , address=145.79.227.23:4545 , piece_id=6MBV4336KPSHR3S67T333ZQCLXX5QBXOYAEBAUG76CVDU2CVDJXA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.374732552 (291h23m15.926203013s) [noise=true , address=45.91.116.12:4225 , piece_id=3OUXQAWMEGWO26XBUBRNAEHXV2DP5EUP5UUKNVRZHHPMASJ57OXQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375180444 (291h23m15.926650906s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375516435 (291h23m15.926986886s) [noise=true , address=145.79.227.24:4581 , piece_id=VVU2M3HLNS4A46SSIQRCOR6WV6J5FUHREINR3TC2H4RJBV4GZNFQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.375888683 (291h23m15.927359144s) [noise=true , address=145.79.227.16:4909 , piece_id=5JKO5HSDEUTJN4TK6CUZ273EYNR6A6Y3625TUUOS3VKHMAPWDHFQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.376543971 (291h23m15.928014432s) [noise=true , address=145.79.227.21:4473 , piece_id=ILRSOD2EDSJXRMVZ7NITZOAZTRQQ5UUW3GKB3QB7DQASVRQVC27A, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.376981635 (291h23m15.928452096s) [noise=true , address=145.79.227.11:4731 , piece_id=CPBGYXE7CIJRVCLWCCMTL22SL2ZSC62KRQINNNACO4MYI4YLRGGQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.377562520 (291h23m15.929032981s) [noise=true , address=145.79.227.31:4799 , piece_id=CHOH62M4JKZGHNHR76M2WUKPE7G772IHGQG7UHYU5PVAYEKC4VPQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.378610416 (291h23m15.930080877s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWQ2CNVJCR5FKJ3OA5MKPH42O27BKLSTIJJIPKEQSDM3EJHNGRQA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.378974038 (291h23m15.930444499s) [noise=true , address=145.79.227.28:4726 , piece_id=72XULSSYQMZQAXFCRQFMXOBRCOE2LILOV6FPDORRPMWI6GVBVGHQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.379921312 (291h23m15.931391773s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.380443847 (291h23m15.931914308s) [noise=true , address=77.90.31.12:4589 , piece_id=S4JENPNCKPBMMRSMBD7AFKWWT6VI2DWM4YWE4O7IPTIZYEBZFOMA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.381260582 (291h23m15.932731044s) [noise=true , address=145.79.227.10:4402 , piece_id=FDDET6WXFVXZJIG2YENY3RVK3SC2LEOSRCR25VLYINEWW5YFJHYQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.381929115 (291h23m15.933399576s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:17.383439992 (291h23m15.934910473s) [noise=true , address=45.91.116.4:4940 , piece_id=OXPGVCFNLUECUPAETAAO6N42JULU6PJQJTXZU4PP6XFA3ENGUFYA, seg_pos={0 0}, upload=256610]: Uploading piece... Nov 21 11:27:18.997259485 (291h23m17.548729936s) [upload=256610]: Next returned nil segment Nov 21 11:27:18.997276468 (291h23m17.548746929s) [upload=256610]: Waiting for error group managing segments... Nov 21 11:27:19.376515178 (291h23m17.927985629s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWQ2CNVJCR5FKJ3OA5MKPH42O27BKLSTIJJIPKEQSDM3EJHNGRQA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.376615770 (291h23m17.928086231s) [noise=true , address=145.79.227.21:4473 , piece_id=ILRSOD2EDSJXRMVZ7NITZOAZTRQQ5UUW3GKB3QB7DQASVRQVC27A, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.380558629 (291h23m17.93202909s) [noise=true , address=145.79.227.11:4731 , piece_id=CPBGYXE7CIJRVCLWCCMTL22SL2ZSC62KRQINNNACO4MYI4YLRGGQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.386618928 (291h23m17.93808942s) [noise=true , address=145.79.227.23:4545 , piece_id=6MBV4336KPSHR3S67T333ZQCLXX5QBXOYAEBAUG76CVDU2CVDJXA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.388538574 (291h23m17.940009035s) [noise=true , address=145.79.227.24:4581 , piece_id=VVU2M3HLNS4A46SSIQRCOR6WV6J5FUHREINR3TC2H4RJBV4GZNFQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.390716079 (291h23m17.94218653s) [noise=true , address=145.79.227.31:4799 , piece_id=CHOH62M4JKZGHNHR76M2WUKPE7G772IHGQG7UHYU5PVAYEKC4VPQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.403575166 (291h23m17.955045627s) [noise=true , address=145.79.227.10:4402 , piece_id=FDDET6WXFVXZJIG2YENY3RVK3SC2LEOSRCR25VLYINEWW5YFJHYQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.433618534 (291h23m17.985088985s) [noise=true , address=145.79.227.28:4726 , piece_id=72XULSSYQMZQAXFCRQFMXOBRCOE2LILOV6FPDORRPMWI6GVBVGHQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.756646770 (291h23m18.308117231s) [noise=true , address=145.79.227.16:4909 , piece_id=5JKO5HSDEUTJN4TK6CUZ273EYNR6A6Y3625TUUOS3VKHMAPWDHFQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.953171670 (291h23m18.504642121s) [noise=true , address=77.90.31.12:4589 , piece_id=S4JENPNCKPBMMRSMBD7AFKWWT6VI2DWM4YWE4O7IPTIZYEBZFOMA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.958723291 (291h23m18.510193762s) [noise=true , address=77.90.31.18:4805 , piece_id=3E62LR5OVVTS3O7CFDRDABMILFBBVE6PS74CXRMTPGC4CVAXNW2A, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.964914050 (291h23m18.516384511s) [noise=true , address=45.91.116.4:4940 , piece_id=OXPGVCFNLUECUPAETAAO6N42JULU6PJQJTXZU4PP6XFA3ENGUFYA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.981247682 (291h23m18.532718144s) [noise=true , address=45.91.116.12:4225 , piece_id=3OUXQAWMEGWO26XBUBRNAEHXV2DP5EUP5UUKNVRZHHPMASJ57OXQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: <nil> Nov 21 11:27:19.981267450 (291h23m18.532737911s) [seg_pos={0 0}, upload=256610]: Segment reached optimal threshold of 13 pieces. Nov 21 11:27:19.981527044 (291h23m18.532997505s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51006->45.91.116.8:4047: use of closed network connection Nov 21 11:27:19.981535420 (291h23m18.533005871s) [noise=true , address=45.91.116.8:4047 , piece_id=HWVGC657LZDVXRP5MSHFD5KWEDXRYAFJB5MYWPWXJF6QHMOI6MJA, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981690315 (291h23m18.533160786s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Nov 21 11:27:19.981717316 (291h23m18.533187767s) [noise=true , address=77.90.31.18:4806 , piece_id=7RLFO2JRULB6S65D2DNWTUQ43PAMKJ2QATS7GUL5G644UD7GOUSQ, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981727796 (291h23m18.533198257s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Nov 21 11:27:19.981735151 (291h23m18.533205602s) [noise=true , address=45.91.116.5:4299 , piece_id=YKATU4LKWE6675KAO57QO5RDWB2IBRXKV5IWKHUQB6PX47PDGQAQ, seg_pos={0 0}, upload=256610]: Long tail cancellation due to optimal threshold Nov 21 11:27:19.981884485 (291h23m18.533354946s) [seg_pos={0 0}, upload=256610]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:27:19.981964216 (291h23m18.533434677s) [upload=256610]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:27:19.981968945 (291h23m18.533439406s) [upload=256610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:27:20.271805188 (291h23m18.82327611s) [upload=256610]: Done uploading segments... Nov 21 11:27:20.271830285 (291h23m18.823300746s) [upload=256610]: Upload finished. err: <nil> Duration: 30.002936371s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 475.057669ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:27:44 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 475.088927ms edge gateway vhost bgp eu1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 416.460159ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:27:40 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 416.487982ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 5.86088002s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:27:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.860908454s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001162817s trace: https://jaeger.dev.storj.io/trace/3dd8e4a3ae37eff6 FAIL (2025-11-21 11:27:36 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.001187504s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 454.261518ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:27:21 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 454.286947ms edge gateway vhost bgp eu1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 389.723909ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:27:14 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 389.752413ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 467.760013ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:27:02 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 467.795048ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 6.962624296s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:27:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 6.962652921s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 386.306957ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/41cfd75a627799ce FAIL (2025-11-21 11:26:58 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 386.34537ms edge linksharing bgp us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 449.598117ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318 trace: https://jaeger.dev.storj.io/trace/5285054b3a776da7 FAIL (2025-11-21 11:26:51 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318": context deadline exceeded Duration: 449.626482ms edge gateway vhost bgp eu1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 407.666502ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace: https://jaeger.dev.storj.io/trace/3e5c193367f9a974 FAIL (2025-11-21 11:26:48 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 407.698975ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 456.127551ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/e2bae25cf885024 FAIL (2025-11-21 11:26:38 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 456.152549ms edge linksharing bgp eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001254863s trace: https://jaeger.dev.storj.io/trace/545070784a71d6c6 FAIL (2025-11-21 11:26:38 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.001283267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 5.29082724s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:26:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 5.290847329s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 1m0.007159647s trace: https://jaeger.dev.storj.io/trace/a7f69ee106efb7f FAIL (2025-11-21 11:26:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== upload=256586]: Uploading piece... Nov 21 11:25:27.210346024 (291h21m25.761816485s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.211425289 (291h21m25.76289575s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213103795 (291h21m25.764574246s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.213408395 (291h21m25.764878856s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.214501747 (291h21m25.765972208s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215013752 (291h21m25.766484213s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.215087572 (291h21m25.766558044s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216432844 (291h21m25.767903305s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.216662281 (291h21m25.768132742s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:27.218008314 (291h21m25.769478775s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Uploading piece... Nov 21 11:25:28.035150976 (291h21m26.586621437s) [upload=256586]: Next returned nil segment Nov 21 11:25:28.035169622 (291h21m26.586640083s) [upload=256586]: Waiting for error group managing segments... Nov 21 11:25:28.416646836 (291h21m26.968117317s) [noise=true , address=185.167.204.206:4064 , piece_id=SRFIWFQLRAGCNIKYLHMGT4QRUL3WH2QPLLIKAJ7DLFJEQUW5DPPA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.417338854 (291h21m26.968809325s) [noise=true , address=185.167.204.205:4033 , piece_id=FEWQVHPIKN3Q6O5AZ6AGJBLUPZ34PBZFB5X3XNSKGPOGI4AIO7JA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.421089406 (291h21m26.972559867s) [noise=true , address=185.167.204.206:4072 , piece_id=GYCO53OWGOMOHK4FHC5GENN4LTQGSZKZKOI5V534RLV3LHLPKC6Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.426252887 (291h21m26.977723348s) [noise=true , address=185.167.204.206:4046 , piece_id=BZ57ZA35ITEOA6CAZJGNODK5MUTXEZWDV7E2OGBZOSMS4OQZYAEA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.431720689 (291h21m26.98319115s) [noise=true , address=185.167.204.205:4027 , piece_id=IVPONKBF2PIHZQAW3L4YA4HOMQV5BQML6SKKCJLIEGT5SZSQ2AOA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.438243951 (291h21m26.989714412s) [noise=true , address=185.167.204.205:4022 , piece_id=BES3YK6PYQV2AKOWSAGHF7DGGWGWSWPIJU2ICAQ5WBHKZ5VT5L6A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.443364331 (291h21m26.994834792s) [noise=true , address=185.167.204.206:4056 , piece_id=EAMZIK6AVJL7IMFF5VOLW3SMCGYKSKB2VXJTNUNLJCVIT7EKXNMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.445594728 (291h21m26.997065179s) [noise=true , address=185.167.204.205:4016 , piece_id=EHQS3546UXRXTL2MGFCSRTPDDPAYXEYLMISPP5AWY337MHX7HFLQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449325822 (291h21m27.000796283s) [noise=true , address=185.167.204.205:4031 , piece_id=ZEI2YSIQC655R7WS43WADXGFT5RKT5LFSN6IZEQ6SHTDVOV7BZBQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.449761211 (291h21m27.001231672s) [noise=true , address=185.167.204.205:4018 , piece_id=7OSLPWJLKFTIJPZEAFMZOAH6JAAMX6HKROI25FODFYBPNJWSMLWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.450419845 (291h21m27.001890306s) [noise=true , address=185.167.204.206:4062 , piece_id=UF3S6NAVGS2KQJPGA5WXP7YG4RQO44P445SNPCWXKXMONZAYUTNQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:28.453201252 (291h21m27.004671713s) [noise=true , address=185.167.204.205:4007 , piece_id=MEBGOPZ6MFMQY4UZARS2LKJKEMNN5YGSLKPJGTWSKS45CUNXSRGQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.661209626 (291h21m28.212680087s) [noise=true , address=89.149.201.147:4016 , piece_id=TGRWGGGBKDANC7YKROWPRMIYNAVFEVE3RXOSUOINFI7KTRS2IYUQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.693169182 (291h21m28.244639643s) [noise=true , address=89.149.201.147:4002 , piece_id=UIYJSCMY3LCLRHZUJB6JFJHHXKMYTVKERTZW7F7X44MWJETPY6XA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.721204497 (291h21m28.272674958s) [noise=true , address=89.149.201.147:4008 , piece_id=HCX7I4ICPEHDWTYDFIQMKZ5EE5CGPQLYAIHF7WZMTUUU6CNZPHTA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.764584711 (291h21m28.316055172s) [noise=true , address=89.149.201.147:4001 , piece_id=QUS6NB3NQBLJFB6YCK3EBDPKCTWEWGKAT3NOR24TVI65SQKHWDOQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.768284075 (291h21m28.319754536s) [noise=true , address=89.149.201.147:4011 , piece_id=ZATNMD5HOK35AWGJJ5V66VDJ4YGUVZQJ2OBERQL32ZV5GW2HBSMQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:29.789681961 (291h21m28.341152422s) [noise=true , address=89.149.201.147:4003 , piece_id=RUAYGN2RGF5JJTWFEGIHWH3BHEUDSS5DN2QJ6YLSHPF5BGUXD3RA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.298598682 (291h21m35.850069143s) [noise=true , address=81.171.14.70:4043 , piece_id=T3NF6JR3VJIGDMPNUBLMJ3AOTQP5IHOGJYORMCSS7ZMOENQETZ7A, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:37.497478356 (291h21m36.048948818s) [noise=true , address=81.171.14.70:4032 , piece_id=CFH2RTR3BLBUB5ZIZCFC6UHT3FHT3B2XSCHQMKXNGDJJGPHEMCTQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:40.406166933 (291h21m38.957637394s) [noise=true , address=81.171.14.70:4045 , piece_id=BRWQ7G3E547PJE2RLGSESIB3HMP5C4M5NSKAJNGWSFXUBEO6SYEQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:25:42.511041157 (291h21m41.062511608s) [noise=true , address=81.171.14.70:4029 , piece_id=DKJM5E7EY7LOSBSEBQOZRC6CI5ZPPE47HFOFQJROPCUMUV567CAA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:12.012682521 (291h22m10.564152982s) [noise=true , address=81.171.14.70:4042 , piece_id=XNREZEUZ5L5KZ3L7ZOL6IRK7IHU6DJN4RLEQZRFMDKDH2JWC24CQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:15.596046283 (291h22m14.147516744s) [noise=true , address=81.171.14.70:4033 , piece_id=JRKPD4EP6FZM5HISOIDJYZY7FAWWQCADAS47FGW2JM5A7RFJR2BQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:18.453059659 (291h22m17.00453012s) [noise=true , address=81.171.14.70:4046 , piece_id=OSONRSF6KJS7F2QJVCC3FPZK4AV7QPG5YSM47CW5LIW7EME44AWA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442583344 (291h22m21.994053805s) [noise=true , address=81.171.14.70:4034 , piece_id=ITRLKYVNFSYBLABNGNFO3HRJNMX3MWB345FIMWSIPBNAVZPA47UA, seg_pos={0 0}, upload=256586]: Done uploading piece. err: <nil> Nov 21 11:26:23.442621967 (291h22m21.994092428s) [seg_pos={0 0}, upload=256586]: Segment reached optimal threshold of 26 pieces. Nov 21 11:26:23.442836976 (291h22m21.994307437s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.442845934 (291h22m21.994316395s) [noise=true , address=81.171.14.70:4044 , piece_id=QYUBNGS2FAFHIWMW7GAGYJ3UVIWVP62LIDCOK6QUXEIQNIXRJDAQ, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443227491 (291h22m21.994697942s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122cYvgTxSvC2aEXnBkRk2o8ARTBXj9gqPoa12vW8hogSN6kpA4): protocol: expected piece hash; context canceled; EOF Nov 21 11:26:23.443237770 (291h22m21.994708231s) [noise=true , address=81.171.14.70:4027 , piece_id=4FDFCCWF3EIROQAFLOHGS743523OYGSXWQSPQEVD3AMQ6WKDQI7Q, seg_pos={0 0}, upload=256586]: Long tail cancellation due to optimal threshold Nov 21 11:26:23.443274720 (291h22m21.994745181s) [seg_pos={0 0}, upload=256586]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Nov 21 11:26:23.443378929 (291h22m21.99484939s) [upload=256586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:26:23.443387044 (291h22m21.994857495s) [upload=256586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:26:23.685139626 (291h22m22.236610077s) [upload=256586]: Done uploading segments... Nov 21 11:26:23.685160676 (291h22m22.236631127s) [upload=256586]: Upload finished. err: <nil> Duration: 1m0.007183251s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...............................X...X.X......X.XX...X.X.X....] duration: 6.04921362s trace: https://jaeger.dev.storj.io/trace/11741d4149da7554 FAIL (2025-11-21 11:26:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21 11:23:32.030021320 (291h19m30.581491781s) [upload=256558]: Starting upload Nov 21 11:23:32.030085402 (291h19m30.581555863s) [upload=256558]: Done starting upload Nov 21 11:23:32.030303317 (291h19m30.581773778s) [upload=256558]: Uploading segments... Nov 21 11:23:32.030356167 (291h19m30.581826628s) [upload=256558]: Got next segment. Inline: false Nov 21 11:23:32.030387106 (291h19m30.581857567s) [upload=256558]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 21 11:23:32.030392215 (291h19m30.581862677s) [upload=256558]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 21 11:23:32.282740232 (291h19m30.834210694s) [seg_pos={0 0}, upload=256558]: Begin upload segment... Nov 21 11:23:32.282810657 (291h19m30.834281118s) [seg_pos={0 0}, upload=256558]: Done begin upload segment. Nov 21 11:23:32.282844210 (291h19m30.834314672s) [noise=true , address=145.79.227.10:4403 , piece_id=IGHC5SMKZFUPDZX67MFAEOG4VENCOZPIAH5JUCITQSXHUXEDKYEQ, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.282946776 (291h19m30.834417267s) [noise=true , address=45.91.116.13:4153 , piece_id=PURNOHU7AO5CU56GSKWHL74CFSNTU5ZQBHSEKZUI7NPZOYYF6NPA, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.283484700 (291h19m30.834955151s) [noise=true , address=145.79.227.9:4365 , piece_id=4KPSLHV2JDV6N2PVMNDDMR5BYQC4BIQCMYS3VB3ZGAKPJYPZRPDA, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.283937994 (291h19m30.835408455s) [noise=true , address=145.79.227.24:4583 , piece_id=JRNBN7B3UTJ553EUKA6YIL6A2GPKPGD3F4Y5YHS3RVXU477KOUKQ, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.284585546 (291h19m30.836056007s) [noise=true , address=145.79.227.14:4838 , piece_id=MYHWTLQ56AWAYXD4RLZBEB36SL5KGG4TEYYOXIRIKTMV65P3YUHA, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.285063186 (291h19m30.836534258s) [noise=true , address=45.91.116.8:4047 , piece_id=EFN6VOXSAOD5WHBCGHMHLF4NYYSJ3ETE6JPA724PRPNCESRKQQWA, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.285553590 (291h19m30.837024051s) [noise=true , address=145.79.227.30:4762 , piece_id=ULANBUXHAJHXOM3TGW6OWZRBWZ355EI6LOHJORJIYQ43S6BIS3OA, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.286099109 (291h19m30.83756957s) [noise=true , address=145.79.227.17:4971 , piece_id=OP3VVWV5OE4T5H464ODJN35XJD23KAGLYSI3XWWKZ4XWUDEOHA2Q, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.287497051 (291h19m30.838967512s) [noise=true , address=145.79.227.32:4836 , piece_id=CU7EW53ZPRHTQJY6XBAQR33GUM4Y2LG7PFTSSWKQGSQZOYIL5C5A, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.287678255 (291h19m30.839148717s) [noise=true , address=45.91.116.13:4154 , piece_id=NHVTVQFG3RACRUG65YEZQWBI2NVYDHCYLLK3JV7FIF7YRDWW2ZLA, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.288943706 (291h19m30.840414167s) [noise=true , address=145.79.227.27:4691 , piece_id=2D3GTJH4RPUJKICOF6TPB73APWHWDXECKKLTIMI3WPCHDEEJ63FA, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.292528432 (291h19m30.843998893s) [noise=true , address=145.79.227.31:4797 , piece_id=MIXKGB3QQIS3A4WQXRZQQKZH2J3M2YIDCQS53JV7PAVHFEEA7ZTA, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.295299748 (291h19m30.846770219s) [noise=true , address=77.90.31.17:4770 , piece_id=YCKFJY26KRRHLEUFDKVDVA4J3VY5UV5ZH6AMPYXFL55V43BTXFJA, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.297553660 (291h19m30.849024121s) [noise=true , address=45.91.116.7:4009 , piece_id=B5NN7NSK25IYA2EYJRPKXA3B5RCXQFFQTS5YENL47SJ6MP3O4VOQ, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.299332547 (291h19m30.850803009s) [noise=true , address=145.79.227.28:4726 , piece_id=ZRK55Y3JWKCWP7VX4KNECZBZ2NSLJDY2EI4BIGSOLEURUHU7UV4A, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:32.301339950 (291h19m30.852810411s) [noise=true , address=145.79.227.20:4437 , piece_id=6XZTGZKSZ6Q6TLFFFSPIWPSHABDO5QMM3NC2VGPFV7N4HREBLSPQ, seg_pos={0 0}, upload=256558]: Uploading piece... Nov 21 11:23:33.894765991 (291h19m32.446236443s) [upload=256558]: Next returned nil segment Nov 21 11:23:33.894811969 (291h19m32.44628243s) [upload=256558]: Waiting for error group managing segments... Nov 21 11:23:34.279788967 (291h19m32.831259428s) [noise=true , address=145.79.227.17:4971 , piece_id=OP3VVWV5OE4T5H464ODJN35XJD23KAGLYSI3XWWKZ4XWUDEOHA2Q, seg_pos={0 0}, upload=256558]: Done uploading piece. err: <nil> Nov 21 11:23:34.280008254 (291h19m32.831478716s) [noise=true , address=145.79.227.9:4365 , piece_id=4KPSLHV2JDV6N2PVMNDDMR5BYQC4BIQCMYS3VB3ZGAKPJYPZRPDA, seg_pos={0 0}, upload=256558]: Done uploading piece. err: <nil> Nov 21 11:23:34.282873009 (291h19m32.83434346s) [noise=true , address=145.79.227.14:4838 , piece_id=MYHWTLQ56AWAYXD4RLZBEB36SL5KGG4TEYYOXIRIKTMV65P3YUHA, seg_pos={0 0}, upload=256558]: Done uploading piece. err: <nil> Nov 21 11:23:34.283363855 (291h19m32.834834316s) [noise=true , address=145.79.227.32:4836 , piece_id=CU7EW53ZPRHTQJY6XBAQR33GUM4Y2LG7PFTSSWKQGSQZOYIL5C5A, seg_pos={0 0}, upload=256558]: Done uploading piece. err: <nil> Nov 21 11:23:34.294748613 (291h19m32.846219074s) [noise=true , address=145.79.227.31:4797 , piece_id=MIXKGB3QQIS3A4WQXRZQQKZH2J3M2YIDCQS53JV7PAVHFEEA7ZTA, seg_pos={0 0}, upload=256558]: Done uploading piece. err: <nil> Nov 21 11:23:34.298848340 (291h19m32.850318801s) [noise=true , address=145.79.227.30:4762 , piece_id=ULANBUXHAJHXOM3TGW6OWZRBWZ355EI6LOHJORJIYQ43S6BIS3OA, seg_pos={0 0}, upload=256558]: Done uploading piece. err: <nil> Nov 21 11:23:34.304011321 (291h19m32.855481772s) [noise=true , address=145.79.227.24:4583 , piece_id=JRNBN7B3UTJ553EUKA6YIL6A2GPKPGD3F4Y5YHS3RVXU477KOUKQ, seg_pos={0 0}, upload=256558]: Done uploading piece. err: <nil> Nov 21 11:23:34.304128554 (291h19m32.855599015s) [noise=true , address=145.79.227.20:4437 , piece_id=6XZTGZKSZ6Q6TLFFFSPIWPSHABDO5QMM3NC2VGPFV7N4HREBLSPQ, seg_pos={0 0}, upload=256558]: Done uploading piece. err: <nil> Nov 21 11:23:34.306480533 (291h19m32.857950994s) [noise=true , address=145.79.227.28:4726 , piece_id=ZRK55Y3JWKCWP7VX4KNECZBZ2NSLJDY2EI4BIGSOLEURUHU7UV4A, seg_pos={0 0}, upload=256558]: Done uploading piece. err: <nil> Nov 21 11:23:34.318824699 (291h19m32.87029516s) [noise=true , address=145.79.227.10:4403 , piece_id=IGHC5SMKZFUPDZX67MFAEOG4VENCOZPIAH5JUCITQSXHUXEDKYEQ, seg_pos={0 0}, upload=256558]: Done uploading piece. err: <nil> Nov 21 11:23:34.319297037 (291h19m32.870767489s) [noise=true , address=145.79.227.27:4691 , piece_id=2D3GTJH4RPUJKICOF6TPB73APWHWDXECKKLTIMI3WPCHDEEJ63FA, seg_pos={0 0}, upload=256558]: Done uploading piece. err: <nil> Nov 21 11:23:34.857536767 (291h19m33.409007228s) [noise=true , address=45.91.116.13:4154 , piece_id=NHVTVQFG3RACRUG65YEZQWBI2NVYDHCYLLK3JV7FIF7YRDWW2ZLA, seg_pos={0 0}, upload=256558]: Done uploading piece. err: <nil> Nov 21 11:23:34.914399273 (291h19m33.465869725s) [noise=true , address=45.91.116.8:4047 , piece_id=EFN6VOXSAOD5WHBCGHMHLF4NYYSJ3ETE6JPA724PRPNCESRKQQWA, seg_pos={0 0}, upload=256558]: Done uploading piece. err: <nil> Nov 21 11:23:34.914418531 (291h19m33.465888992s) [seg_pos={0 0}, upload=256558]: Segment reached optimal threshold of 13 pieces. Nov 21 11:23:34.914651644 (291h19m33.466122105s) [noise=true , address=45.91.116.7:4009 , piece_id=B5NN7NSK25IYA2EYJRPKXA3B5RCXQFFQTS5YENL47SJ6MP3O4VOQ, seg_pos={0 0}, upload=256558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:47920->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:47920->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47920->45.91.116.7:4009: use of closed network connection Nov 21 11:23:34.914661483 (291h19m33.466131934s) [noise=true , address=45.91.116.7:4009 , piece_id=B5NN7NSK25IYA2EYJRPKXA3B5RCXQFFQTS5YENL47SJ6MP3O4VOQ, seg_pos={0 0}, upload=256558]: Long tail cancellation due to optimal threshold Nov 21 11:23:34.914721086 (291h19m33.466191537s) [noise=true , address=45.91.116.13:4153 , piece_id=PURNOHU7AO5CU56GSKWHL74CFSNTU5ZQBHSEKZUI7NPZOYYF6NPA, seg_pos={0 0}, upload=256558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Nov 21 11:23:34.914726748 (291h19m33.466197209s) [noise=true , address=45.91.116.13:4153 , piece_id=PURNOHU7AO5CU56GSKWHL74CFSNTU5ZQBHSEKZUI7NPZOYYF6NPA, seg_pos={0 0}, upload=256558]: Long tail cancellation due to optimal threshold Nov 21 11:23:34.914893925 (291h19m33.466364376s) [noise=true , address=77.90.31.17:4770 , piece_id=YCKFJY26KRRHLEUFDKVDVA4J3VY5UV5ZH6AMPYXFL55V43BTXFJA, seg_pos={0 0}, upload=256558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Nov 21 11:23:34.914908684 (291h19m33.466379145s) [noise=true , address=77.90.31.17:4770 , piece_id=YCKFJY26KRRHLEUFDKVDVA4J3VY5UV5ZH6AMPYXFL55V43BTXFJA, seg_pos={0 0}, upload=256558]: Long tail cancellation due to optimal threshold Nov 21 11:23:34.914925936 (291h19m33.466396397s) [seg_pos={0 0}, upload=256558]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 21 11:23:34.915009837 (291h19m33.466480288s) [upload=256558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 21 11:23:34.915014776 (291h19m33.466485237s) [upload=256558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 21 11:23:35.212831524 (291h19m33.764301985s) [upload=256558]: Done uploading segments... Nov 21 11:23:35.212856983 (291h19m33.764327434s) [upload=256558]: Upload finished. err: <nil> Duration: 6.049243377s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)