Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.......X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X.] duration: 4.169163326s trace: https://jaeger.dev.storj.io/trace/7c81e63b0bcc6d75 FAIL (2026-04-17 22:22:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.169189174s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.....................................XXXXX..............XXXX] duration: 1m0.002265994s trace: https://jaeger.dev.storj.io/trace/6a7e45e61a3c62fc FAIL (2026-04-17 22:22:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002281354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X..........X...] duration: 9.501303572s trace: https://jaeger.dev.storj.io/trace/56f1ff2cba8d586e FAIL (2026-04-17 22:22:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12745008s) [upload=236716]: Uploading segments... Apr 17 22:20:15.093601222 (246h34m1.912837246s) [upload=236716]: Got next segment. Inline: false Apr 17 22:20:15.093646419 (246h34m1.912882443s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:20:15.093652220 (246h34m1.912888254s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:20:15.327751532 (246h34m2.146987556s) [seg_pos={0 0}, upload=236716]: Begin upload segment... Apr 17 22:20:15.327797871 (246h34m2.147033895s) [seg_pos={0 0}, upload=236716]: Done begin upload segment. Apr 17 22:20:15.327815896 (246h34m2.14705192s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.327866914 (246h34m2.147102938s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.328609949 (246h34m2.147845973s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329005071 (246h34m2.148241095s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329166012 (246h34m2.148402036s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329497020 (246h34m2.148733044s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329541275 (246h34m2.148777299s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329818652 (246h34m2.149054676s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330271054 (246h34m2.149507078s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330620629 (246h34m2.149856653s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331112438 (246h34m2.150348462s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331548820 (246h34m2.150784854s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331925899 (246h34m2.151161923s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.332558900 (246h34m2.151794924s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.333059175 (246h34m2.152295199s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.335146704 (246h34m2.154382738s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:17.153184874 (246h34m3.972420898s) [upload=236716]: Next returned nil segment Apr 17 22:20:17.153196477 (246h34m3.972432511s) [upload=236716]: Waiting for error group managing segments... Apr 17 22:20:17.454914848 (246h34m4.274150872s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:17.853046821 (246h34m4.672282845s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.277080214 (246h34m5.096316228s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.358624359 (246h34m5.177860383s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.511842111 (246h34m5.331078135s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.649787067 (246h34m5.469023091s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783084333 (246h34m5.602320357s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783561244 (246h34m5.602797258s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.956713800 (246h34m5.775949875s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.037983717 (246h34m5.857219751s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.790697537 (246h34m6.609933551s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.891465144 (246h34m6.710701168s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266765615 (246h34m7.086001639s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266784993 (246h34m7.086021027s) [seg_pos={0 0}, upload=236716]: Satisfied cohort requirements after 13 pieces Apr 17 22:20:20.267040255 (246h34m7.086276279s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW): protocol: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection Apr 17 22:20:20.267048020 (246h34m7.086284044s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267107014 (246h34m7.086343038s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG): protocol: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection Apr 17 22:20:20.267111904 (246h34m7.086347928s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267214120 (246h34m7.086450144s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Apr 17 22:20:20.267220934 (246h34m7.086456958s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267234309 (246h34m7.086470333s) [seg_pos={0 0}, upload=236716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:20:20.267293925 (246h34m7.086529949s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:20:20.267298444 (246h34m7.086534468s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:20:20.553057934 (246h34m7.372293958s) [upload=236716]: Done uploading segments... Apr 17 22:20:20.553075518 (246h34m7.372311542s) [upload=236716]: Upload finished. err: <nil> Duration: 9.501327838s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [........X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..X] duration: 5.001093964s trace: https://jaeger.dev.storj.io/trace/7c81e63b0bcc6d75 FAIL (2026-04-17 22:21:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001116777s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X..........X..] duration: 9.557344091s trace: https://jaeger.dev.storj.io/trace/56f1ff2cba8d586e FAIL (2026-04-17 22:21:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12745008s) [upload=236716]: Uploading segments... Apr 17 22:20:15.093601222 (246h34m1.912837246s) [upload=236716]: Got next segment. Inline: false Apr 17 22:20:15.093646419 (246h34m1.912882443s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:20:15.093652220 (246h34m1.912888254s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:20:15.327751532 (246h34m2.146987556s) [seg_pos={0 0}, upload=236716]: Begin upload segment... Apr 17 22:20:15.327797871 (246h34m2.147033895s) [seg_pos={0 0}, upload=236716]: Done begin upload segment. Apr 17 22:20:15.327815896 (246h34m2.14705192s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.327866914 (246h34m2.147102938s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.328609949 (246h34m2.147845973s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329005071 (246h34m2.148241095s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329166012 (246h34m2.148402036s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329497020 (246h34m2.148733044s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329541275 (246h34m2.148777299s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329818652 (246h34m2.149054676s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330271054 (246h34m2.149507078s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330620629 (246h34m2.149856653s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331112438 (246h34m2.150348462s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331548820 (246h34m2.150784854s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331925899 (246h34m2.151161923s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.332558900 (246h34m2.151794924s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.333059175 (246h34m2.152295199s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.335146704 (246h34m2.154382738s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:17.153184874 (246h34m3.972420898s) [upload=236716]: Next returned nil segment Apr 17 22:20:17.153196477 (246h34m3.972432511s) [upload=236716]: Waiting for error group managing segments... Apr 17 22:20:17.454914848 (246h34m4.274150872s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:17.853046821 (246h34m4.672282845s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.277080214 (246h34m5.096316228s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.358624359 (246h34m5.177860383s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.511842111 (246h34m5.331078135s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.649787067 (246h34m5.469023091s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783084333 (246h34m5.602320357s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783561244 (246h34m5.602797258s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.956713800 (246h34m5.775949875s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.037983717 (246h34m5.857219751s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.790697537 (246h34m6.609933551s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.891465144 (246h34m6.710701168s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266765615 (246h34m7.086001639s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266784993 (246h34m7.086021027s) [seg_pos={0 0}, upload=236716]: Satisfied cohort requirements after 13 pieces Apr 17 22:20:20.267040255 (246h34m7.086276279s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW): protocol: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection Apr 17 22:20:20.267048020 (246h34m7.086284044s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267107014 (246h34m7.086343038s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG): protocol: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection Apr 17 22:20:20.267111904 (246h34m7.086347928s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267214120 (246h34m7.086450144s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Apr 17 22:20:20.267220934 (246h34m7.086456958s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267234309 (246h34m7.086470333s) [seg_pos={0 0}, upload=236716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:20:20.267293925 (246h34m7.086529949s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:20:20.267298444 (246h34m7.086534468s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:20:20.553057934 (246h34m7.372293958s) [upload=236716]: Done uploading segments... Apr 17 22:20:20.553075518 (246h34m7.372311542s) [upload=236716]: Upload finished. err: <nil> Duration: 9.557374671s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.........X............X.XX...X.......XXXX.X.XX.XX.XXXX...X..] duration: 5.000715004s trace: https://jaeger.dev.storj.io/trace/281ef638e16750f7 FAIL (2026-04-17 22:21:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000738239s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......................................XXXXX..............XXX] duration: 1m0.00077685s trace: https://jaeger.dev.storj.io/trace/7e397d6b39319fe1 FAIL (2026-04-17 22:21:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000799695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X..........X.] duration: 8.41249953s trace: https://jaeger.dev.storj.io/trace/56f1ff2cba8d586e FAIL (2026-04-17 22:21:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12745008s) [upload=236716]: Uploading segments... Apr 17 22:20:15.093601222 (246h34m1.912837246s) [upload=236716]: Got next segment. Inline: false Apr 17 22:20:15.093646419 (246h34m1.912882443s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:20:15.093652220 (246h34m1.912888254s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:20:15.327751532 (246h34m2.146987556s) [seg_pos={0 0}, upload=236716]: Begin upload segment... Apr 17 22:20:15.327797871 (246h34m2.147033895s) [seg_pos={0 0}, upload=236716]: Done begin upload segment. Apr 17 22:20:15.327815896 (246h34m2.14705192s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.327866914 (246h34m2.147102938s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.328609949 (246h34m2.147845973s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329005071 (246h34m2.148241095s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329166012 (246h34m2.148402036s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329497020 (246h34m2.148733044s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329541275 (246h34m2.148777299s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329818652 (246h34m2.149054676s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330271054 (246h34m2.149507078s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330620629 (246h34m2.149856653s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331112438 (246h34m2.150348462s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331548820 (246h34m2.150784854s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331925899 (246h34m2.151161923s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.332558900 (246h34m2.151794924s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.333059175 (246h34m2.152295199s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.335146704 (246h34m2.154382738s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:17.153184874 (246h34m3.972420898s) [upload=236716]: Next returned nil segment Apr 17 22:20:17.153196477 (246h34m3.972432511s) [upload=236716]: Waiting for error group managing segments... Apr 17 22:20:17.454914848 (246h34m4.274150872s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:17.853046821 (246h34m4.672282845s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.277080214 (246h34m5.096316228s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.358624359 (246h34m5.177860383s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.511842111 (246h34m5.331078135s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.649787067 (246h34m5.469023091s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783084333 (246h34m5.602320357s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783561244 (246h34m5.602797258s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.956713800 (246h34m5.775949875s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.037983717 (246h34m5.857219751s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.790697537 (246h34m6.609933551s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.891465144 (246h34m6.710701168s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266765615 (246h34m7.086001639s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266784993 (246h34m7.086021027s) [seg_pos={0 0}, upload=236716]: Satisfied cohort requirements after 13 pieces Apr 17 22:20:20.267040255 (246h34m7.086276279s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW): protocol: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection Apr 17 22:20:20.267048020 (246h34m7.086284044s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267107014 (246h34m7.086343038s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG): protocol: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection Apr 17 22:20:20.267111904 (246h34m7.086347928s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267214120 (246h34m7.086450144s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Apr 17 22:20:20.267220934 (246h34m7.086456958s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267234309 (246h34m7.086470333s) [seg_pos={0 0}, upload=236716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:20:20.267293925 (246h34m7.086529949s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:20:20.267298444 (246h34m7.086534468s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:20:20.553057934 (246h34m7.372293958s) [upload=236716]: Done uploading segments... Apr 17 22:20:20.553075518 (246h34m7.372311542s) [upload=236716]: Upload finished. err: <nil> Duration: 8.412528136s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [..........X............X.XX...X.......XXXX.X.XX.XX.XXXX...X.] duration: 4.73454751s trace: https://jaeger.dev.storj.io/trace/281ef638e16750f7 FAIL (2026-04-17 22:20:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.734603999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X..........X] duration: 30.002871816s trace: https://jaeger.dev.storj.io/trace/56f1ff2cba8d586e FAIL (2026-04-17 22:20:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12745008s) [upload=236716]: Uploading segments... Apr 17 22:20:15.093601222 (246h34m1.912837246s) [upload=236716]: Got next segment. Inline: false Apr 17 22:20:15.093646419 (246h34m1.912882443s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:20:15.093652220 (246h34m1.912888254s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:20:15.327751532 (246h34m2.146987556s) [seg_pos={0 0}, upload=236716]: Begin upload segment... Apr 17 22:20:15.327797871 (246h34m2.147033895s) [seg_pos={0 0}, upload=236716]: Done begin upload segment. Apr 17 22:20:15.327815896 (246h34m2.14705192s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.327866914 (246h34m2.147102938s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.328609949 (246h34m2.147845973s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329005071 (246h34m2.148241095s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329166012 (246h34m2.148402036s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329497020 (246h34m2.148733044s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329541275 (246h34m2.148777299s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.329818652 (246h34m2.149054676s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330271054 (246h34m2.149507078s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.330620629 (246h34m2.149856653s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331112438 (246h34m2.150348462s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331548820 (246h34m2.150784854s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.331925899 (246h34m2.151161923s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.332558900 (246h34m2.151794924s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.333059175 (246h34m2.152295199s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:15.335146704 (246h34m2.154382738s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Uploading piece... Apr 17 22:20:17.153184874 (246h34m3.972420898s) [upload=236716]: Next returned nil segment Apr 17 22:20:17.153196477 (246h34m3.972432511s) [upload=236716]: Waiting for error group managing segments... Apr 17 22:20:17.454914848 (246h34m4.274150872s) [noise=true , address=77.90.31.14:4661 , piece_id=DFYPRPKPNAPDSKM3GBM6IBZ4ZXXXTUQSX2RTIZV2KY5PW5VO7P4A, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:17.853046821 (246h34m4.672282845s) [noise=true , address=45.91.116.10:4085 , piece_id=VIGNR4ZOWDR5YMSZCQNAY7NHUUDOVXLBKZB7OIYLBDCKXMYAOJWQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.277080214 (246h34m5.096316228s) [noise=true , address=145.79.227.8:4334 , piece_id=AWZ3FOFRCMP5B5AJI2UK3W6EPEAR57ZCAZAIURVUJBWBIJTIWYKA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.358624359 (246h34m5.177860383s) [noise=true , address=45.91.116.4:4953 , piece_id=DBKQYZZ35JZ56GTNGVRBFNBD5DYYEWHKCGC6EV7ED7VDYXNODSVA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.511842111 (246h34m5.331078135s) [noise=true , address=45.91.116.7:4026 , piece_id=MEPEQ3YF3AKJ7DFNXPTMFBERNYN26P2IJQJTFFTGCKPI6O6DDSTQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.649787067 (246h34m5.469023091s) [noise=true , address=45.91.116.11:4118 , piece_id=7M6ZSER2JPAXFJYYN3IUU7O2YSQAR2C4KPTPBQZIEEIJJA2HQ2CA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783084333 (246h34m5.602320357s) [noise=true , address=45.91.116.9:4263 , piece_id=C6A5TVZ3RH2BR5JHBIEQLR27PDCUVLSWCMHMKBFW3T6BJGTRPMUQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.783561244 (246h34m5.602797258s) [noise=true , address=145.79.227.26:4656 , piece_id=PQZKAF7U3EJSMJTZX5YNFYFL4OHKG5K7ZG7RV4LOMRESJDZCXJCQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:18.956713800 (246h34m5.775949875s) [noise=true , address=145.79.227.50:4378 , piece_id=DLD3CPOPGU4ZNHZNAQTV5CZKYLFMUML2GFPLZ6SA7RAHVK4UCWZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.037983717 (246h34m5.857219751s) [noise=true , address=145.79.227.24:4583 , piece_id=PZ5SMGLDCJYZWEUQD2TDOL7C4OLHZUDOCKXKBRS2KHRU5K2DW2WA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.790697537 (246h34m6.609933551s) [noise=true , address=145.79.227.29:4941 , piece_id=EZFV64BXZ5WKPNBKBYFM5XXZYW7Q57S6T3RMPZKKZ6FUOR25Q3RQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:19.891465144 (246h34m6.710701168s) [noise=true , address=145.79.227.30:4761 , piece_id=GP7R7DGLI6SKFPSH43FFP6K5FVLYPXRF7PHFIJUJV4QFJVHXSUBQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266765615 (246h34m7.086001639s) [noise=true , address=145.79.227.47:4268 , piece_id=3UNKMHCHTP5RVQ2XTPVYMDEUUS2NQUKHBTLPFMNZ5KSAONLKQPEQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: <nil> Apr 17 22:20:20.266784993 (246h34m7.086021027s) [seg_pos={0 0}, upload=236716]: Satisfied cohort requirements after 13 pieces Apr 17 22:20:20.267040255 (246h34m7.086276279s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW): protocol: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40438->145.79.227.2:4119: use of closed network connection Apr 17 22:20:20.267048020 (246h34m7.086284044s) [noise=true , address=145.79.227.2:4119 , piece_id=XZ6PVZVLRDK6DISWZQ4FB34QRAWETFUGJAINHPLYWHCFTMXDK5EQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267107014 (246h34m7.086343038s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG): protocol: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37660->145.79.227.38:4982: use of closed network connection Apr 17 22:20:20.267111904 (246h34m7.086347928s) [noise=true , address=145.79.227.38:4982 , piece_id=GQ6VSY7OMGM6AC6NX2STDPKJXXQ4RYYJPIRSVKW2H26XTYLTSGZA, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267214120 (246h34m7.086450144s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Apr 17 22:20:20.267220934 (246h34m7.086456958s) [noise=true , address=145.79.227.15:4874 , piece_id=27ZSSFFMRYPBALRFAE67O56PGDCKVDFDBPW5EN3ATVT7IN6LWILQ, seg_pos={0 0}, upload=236716]: Long tail cancellation due to optimal threshold Apr 17 22:20:20.267234309 (246h34m7.086470333s) [seg_pos={0 0}, upload=236716]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:20:20.267293925 (246h34m7.086529949s) [upload=236716]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:20:20.267298444 (246h34m7.086534468s) [upload=236716]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:20:20.553057934 (246h34m7.372293958s) [upload=236716]: Done uploading segments... Apr 17 22:20:20.553075518 (246h34m7.372311542s) [upload=236716]: Upload finished. err: <nil> Duration: 30.002892967s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.......................................XXXXX..............XX] duration: 1m0.00083783s trace: https://jaeger.dev.storj.io/trace/f85cf54b682d8b6 FAIL (2026-04-17 22:20:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00084788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [...........X............X.XX...X.......XXXX.X.XX.XX.XXXX...X] duration: 5.001714438s trace: https://jaeger.dev.storj.io/trace/281ef638e16750f7 FAIL (2026-04-17 22:19:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001736572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 11.095165666s trace: https://jaeger.dev.storj.io/trace/5cc52839840193df FAIL (2026-04-17 22:19:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:11:17.642192354 (246h25m4.461428378s) [upload=236583]: Starting upload Apr 17 22:11:17.642264764 (246h25m4.461500788s) [upload=236583]: Done starting upload Apr 17 22:11:17.642342734 (246h25m4.461578758s) [upload=236583]: Uploading segments... Apr 17 22:11:17.642525718 (246h25m4.461761742s) [upload=236583]: Got next segment. Inline: false Apr 17 22:11:17.642558962 (246h25m4.461794986s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:11:17.642563912 (246h25m4.461799936s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:11:17.882374413 (246h25m4.701610437s) [seg_pos={0 0}, upload=236583]: Begin upload segment... Apr 17 22:11:17.882438487 (246h25m4.701674511s) [seg_pos={0 0}, upload=236583]: Done begin upload segment. Apr 17 22:11:17.885163417 (246h25m4.704399471s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.890822067 (246h25m4.710058101s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.891352060 (246h25m4.710588084s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.893193865 (246h25m4.712429889s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895445962 (246h25m4.714681986s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895518891 (246h25m4.714754915s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.897292695 (246h25m4.716528719s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.899077650 (246h25m4.718313674s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900145952 (246h25m4.719381976s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900530905 (246h25m4.719766929s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.901710130 (246h25m4.720946154s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.902279129 (246h25m4.721515153s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903290060 (246h25m4.722526084s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903897372 (246h25m4.723133396s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.904735619 (246h25m4.723971643s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.905349224 (246h25m4.724585248s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:19.706338743 (246h25m6.525574767s) [upload=236583]: Next returned nil segment Apr 17 22:11:19.706356788 (246h25m6.525592812s) [upload=236583]: Waiting for error group managing segments... Apr 17 22:11:19.994408492 (246h25m6.813644527s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:19.995247402 (246h25m6.814483426s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.782393945 (246h25m7.601629969s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.929683807 (246h25m7.748919831s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.038934967 (246h25m7.858170991s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.351460856 (246h25m8.17069688s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.445533741 (246h25m8.264769765s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.481223641 (246h25m8.300459665s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.603836618 (246h25m8.423072642s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.713343271 (246h25m8.532579295s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.729259759 (246h25m8.548495783s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.107132825 (246h25m8.926368849s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189431728 (246h25m9.008667762s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189455404 (246h25m9.008691428s) [seg_pos={0 0}, upload=236583]: Satisfied cohort requirements after 13 pieces Apr 17 22:11:22.189633718 (246h25m9.008869752s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189642284 (246h25m9.008878308s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189645831 (246h25m9.008881865s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189656893 (246h25m9.008892937s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189838644 (246h25m9.009074668s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189844715 (246h25m9.009080739s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189861708 (246h25m9.009097742s) [seg_pos={0 0}, upload=236583]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:11:22.189948866 (246h25m9.00918489s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:11:22.189954657 (246h25m9.009190691s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:11:22.483797551 (246h25m9.303033585s) [upload=236583]: Done uploading segments... Apr 17 22:11:22.483827999 (246h25m9.303064023s) [upload=236583]: Upload finished. err: <nil> Duration: 11.095196196s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [X.......................................XXXXX..............X] duration: 1m0.001420791s trace: https://jaeger.dev.storj.io/trace/74d1757f48f20477 FAIL (2026-04-17 22:19:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001448196s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 7.788300426s trace: https://jaeger.dev.storj.io/trace/5cc52839840193df FAIL (2026-04-17 22:18:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:11:17.642192354 (246h25m4.461428378s) [upload=236583]: Starting upload Apr 17 22:11:17.642264764 (246h25m4.461500788s) [upload=236583]: Done starting upload Apr 17 22:11:17.642342734 (246h25m4.461578758s) [upload=236583]: Uploading segments... Apr 17 22:11:17.642525718 (246h25m4.461761742s) [upload=236583]: Got next segment. Inline: false Apr 17 22:11:17.642558962 (246h25m4.461794986s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:11:17.642563912 (246h25m4.461799936s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:11:17.882374413 (246h25m4.701610437s) [seg_pos={0 0}, upload=236583]: Begin upload segment... Apr 17 22:11:17.882438487 (246h25m4.701674511s) [seg_pos={0 0}, upload=236583]: Done begin upload segment. Apr 17 22:11:17.885163417 (246h25m4.704399471s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.890822067 (246h25m4.710058101s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.891352060 (246h25m4.710588084s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.893193865 (246h25m4.712429889s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895445962 (246h25m4.714681986s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895518891 (246h25m4.714754915s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.897292695 (246h25m4.716528719s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.899077650 (246h25m4.718313674s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900145952 (246h25m4.719381976s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900530905 (246h25m4.719766929s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.901710130 (246h25m4.720946154s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.902279129 (246h25m4.721515153s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903290060 (246h25m4.722526084s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903897372 (246h25m4.723133396s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.904735619 (246h25m4.723971643s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.905349224 (246h25m4.724585248s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:19.706338743 (246h25m6.525574767s) [upload=236583]: Next returned nil segment Apr 17 22:11:19.706356788 (246h25m6.525592812s) [upload=236583]: Waiting for error group managing segments... Apr 17 22:11:19.994408492 (246h25m6.813644527s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:19.995247402 (246h25m6.814483426s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.782393945 (246h25m7.601629969s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.929683807 (246h25m7.748919831s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.038934967 (246h25m7.858170991s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.351460856 (246h25m8.17069688s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.445533741 (246h25m8.264769765s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.481223641 (246h25m8.300459665s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.603836618 (246h25m8.423072642s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.713343271 (246h25m8.532579295s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.729259759 (246h25m8.548495783s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.107132825 (246h25m8.926368849s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189431728 (246h25m9.008667762s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189455404 (246h25m9.008691428s) [seg_pos={0 0}, upload=236583]: Satisfied cohort requirements after 13 pieces Apr 17 22:11:22.189633718 (246h25m9.008869752s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189642284 (246h25m9.008878308s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189645831 (246h25m9.008881865s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189656893 (246h25m9.008892937s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189838644 (246h25m9.009074668s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189844715 (246h25m9.009080739s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189861708 (246h25m9.009097742s) [seg_pos={0 0}, upload=236583]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:11:22.189948866 (246h25m9.00918489s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:11:22.189954657 (246h25m9.009190691s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:11:22.483797551 (246h25m9.303033585s) [upload=236583]: Done uploading segments... Apr 17 22:11:22.483827999 (246h25m9.303064023s) [upload=236583]: Upload finished. err: <nil> Duration: 7.788321627s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [............X............X.XX...X.......XXXX.X.XX.XX.XXXX...] duration: 4.65357566s trace: https://jaeger.dev.storj.io/trace/27efa76c186e6633 FAIL (2026-04-17 22:18:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.653594527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 9.133501481s trace: https://jaeger.dev.storj.io/trace/5cc52839840193df FAIL (2026-04-17 22:18:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:11:17.642192354 (246h25m4.461428378s) [upload=236583]: Starting upload Apr 17 22:11:17.642264764 (246h25m4.461500788s) [upload=236583]: Done starting upload Apr 17 22:11:17.642342734 (246h25m4.461578758s) [upload=236583]: Uploading segments... Apr 17 22:11:17.642525718 (246h25m4.461761742s) [upload=236583]: Got next segment. Inline: false Apr 17 22:11:17.642558962 (246h25m4.461794986s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:11:17.642563912 (246h25m4.461799936s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:11:17.882374413 (246h25m4.701610437s) [seg_pos={0 0}, upload=236583]: Begin upload segment... Apr 17 22:11:17.882438487 (246h25m4.701674511s) [seg_pos={0 0}, upload=236583]: Done begin upload segment. Apr 17 22:11:17.885163417 (246h25m4.704399471s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.890822067 (246h25m4.710058101s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.891352060 (246h25m4.710588084s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.893193865 (246h25m4.712429889s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895445962 (246h25m4.714681986s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895518891 (246h25m4.714754915s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.897292695 (246h25m4.716528719s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.899077650 (246h25m4.718313674s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900145952 (246h25m4.719381976s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900530905 (246h25m4.719766929s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.901710130 (246h25m4.720946154s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.902279129 (246h25m4.721515153s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903290060 (246h25m4.722526084s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903897372 (246h25m4.723133396s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.904735619 (246h25m4.723971643s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.905349224 (246h25m4.724585248s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:19.706338743 (246h25m6.525574767s) [upload=236583]: Next returned nil segment Apr 17 22:11:19.706356788 (246h25m6.525592812s) [upload=236583]: Waiting for error group managing segments... Apr 17 22:11:19.994408492 (246h25m6.813644527s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:19.995247402 (246h25m6.814483426s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.782393945 (246h25m7.601629969s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.929683807 (246h25m7.748919831s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.038934967 (246h25m7.858170991s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.351460856 (246h25m8.17069688s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.445533741 (246h25m8.264769765s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.481223641 (246h25m8.300459665s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.603836618 (246h25m8.423072642s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.713343271 (246h25m8.532579295s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.729259759 (246h25m8.548495783s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.107132825 (246h25m8.926368849s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189431728 (246h25m9.008667762s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189455404 (246h25m9.008691428s) [seg_pos={0 0}, upload=236583]: Satisfied cohort requirements after 13 pieces Apr 17 22:11:22.189633718 (246h25m9.008869752s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189642284 (246h25m9.008878308s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189645831 (246h25m9.008881865s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189656893 (246h25m9.008892937s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189838644 (246h25m9.009074668s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189844715 (246h25m9.009080739s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189861708 (246h25m9.009097742s) [seg_pos={0 0}, upload=236583]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:11:22.189948866 (246h25m9.00918489s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:11:22.189954657 (246h25m9.009190691s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:11:22.483797551 (246h25m9.303033585s) [upload=236583]: Done uploading segments... Apr 17 22:11:22.483827999 (246h25m9.303064023s) [upload=236583]: Upload finished. err: <nil> Duration: 9.133524816s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.............X............X.XX...X.......XXXX.X.XX.XX.XXXX..] duration: 4.404796125s trace: https://jaeger.dev.storj.io/trace/27efa76c186e6633 FAIL (2026-04-17 22:18:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.405023324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 9.21009897s trace: https://jaeger.dev.storj.io/trace/5cc52839840193df FAIL (2026-04-17 22:17:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:11:17.642192354 (246h25m4.461428378s) [upload=236583]: Starting upload Apr 17 22:11:17.642264764 (246h25m4.461500788s) [upload=236583]: Done starting upload Apr 17 22:11:17.642342734 (246h25m4.461578758s) [upload=236583]: Uploading segments... Apr 17 22:11:17.642525718 (246h25m4.461761742s) [upload=236583]: Got next segment. Inline: false Apr 17 22:11:17.642558962 (246h25m4.461794986s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:11:17.642563912 (246h25m4.461799936s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:11:17.882374413 (246h25m4.701610437s) [seg_pos={0 0}, upload=236583]: Begin upload segment... Apr 17 22:11:17.882438487 (246h25m4.701674511s) [seg_pos={0 0}, upload=236583]: Done begin upload segment. Apr 17 22:11:17.885163417 (246h25m4.704399471s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.890822067 (246h25m4.710058101s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.891352060 (246h25m4.710588084s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.893193865 (246h25m4.712429889s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895445962 (246h25m4.714681986s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895518891 (246h25m4.714754915s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.897292695 (246h25m4.716528719s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.899077650 (246h25m4.718313674s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900145952 (246h25m4.719381976s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900530905 (246h25m4.719766929s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.901710130 (246h25m4.720946154s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.902279129 (246h25m4.721515153s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903290060 (246h25m4.722526084s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903897372 (246h25m4.723133396s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.904735619 (246h25m4.723971643s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.905349224 (246h25m4.724585248s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:19.706338743 (246h25m6.525574767s) [upload=236583]: Next returned nil segment Apr 17 22:11:19.706356788 (246h25m6.525592812s) [upload=236583]: Waiting for error group managing segments... Apr 17 22:11:19.994408492 (246h25m6.813644527s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:19.995247402 (246h25m6.814483426s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.782393945 (246h25m7.601629969s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.929683807 (246h25m7.748919831s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.038934967 (246h25m7.858170991s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.351460856 (246h25m8.17069688s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.445533741 (246h25m8.264769765s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.481223641 (246h25m8.300459665s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.603836618 (246h25m8.423072642s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.713343271 (246h25m8.532579295s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.729259759 (246h25m8.548495783s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.107132825 (246h25m8.926368849s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189431728 (246h25m9.008667762s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189455404 (246h25m9.008691428s) [seg_pos={0 0}, upload=236583]: Satisfied cohort requirements after 13 pieces Apr 17 22:11:22.189633718 (246h25m9.008869752s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189642284 (246h25m9.008878308s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189645831 (246h25m9.008881865s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189656893 (246h25m9.008892937s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189838644 (246h25m9.009074668s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189844715 (246h25m9.009080739s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189861708 (246h25m9.009097742s) [seg_pos={0 0}, upload=236583]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:11:22.189948866 (246h25m9.00918489s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:11:22.189954657 (246h25m9.009190691s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:11:22.483797551 (246h25m9.303033585s) [upload=236583]: Done uploading segments... Apr 17 22:11:22.483827999 (246h25m9.303064023s) [upload=236583]: Upload finished. err: <nil> Duration: 9.210113958s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.X.......................................XXXXX..............] duration: 16.060863334s trace: https://jaeger.dev.storj.io/trace/1999777fac8036ca FAIL (2026-04-17 22:17:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.060900476s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..............X............X.XX...X.......XXXX.X.XX.XX.XXXX.] duration: 4.848540872s trace: https://jaeger.dev.storj.io/trace/27efa76c186e6633 FAIL (2026-04-17 22:17:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.848557694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..X.......................................XXXXX.............] duration: 35.558062188s trace: https://jaeger.dev.storj.io/trace/1999777fac8036ca FAIL (2026-04-17 22:17:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.558087928s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...............X............X.XX...X.......XXXX.X.XX.XX.XXXX] duration: 5.000630611s trace: https://jaeger.dev.storj.io/trace/27efa76c186e6633 FAIL (2026-04-17 22:17:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000659947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 8.442995423s trace: https://jaeger.dev.storj.io/trace/5cc52839840193df FAIL (2026-04-17 22:16:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:11:17.642192354 (246h25m4.461428378s) [upload=236583]: Starting upload Apr 17 22:11:17.642264764 (246h25m4.461500788s) [upload=236583]: Done starting upload Apr 17 22:11:17.642342734 (246h25m4.461578758s) [upload=236583]: Uploading segments... Apr 17 22:11:17.642525718 (246h25m4.461761742s) [upload=236583]: Got next segment. Inline: false Apr 17 22:11:17.642558962 (246h25m4.461794986s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:11:17.642563912 (246h25m4.461799936s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:11:17.882374413 (246h25m4.701610437s) [seg_pos={0 0}, upload=236583]: Begin upload segment... Apr 17 22:11:17.882438487 (246h25m4.701674511s) [seg_pos={0 0}, upload=236583]: Done begin upload segment. Apr 17 22:11:17.885163417 (246h25m4.704399471s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.890822067 (246h25m4.710058101s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.891352060 (246h25m4.710588084s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.893193865 (246h25m4.712429889s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895445962 (246h25m4.714681986s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895518891 (246h25m4.714754915s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.897292695 (246h25m4.716528719s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.899077650 (246h25m4.718313674s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900145952 (246h25m4.719381976s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900530905 (246h25m4.719766929s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.901710130 (246h25m4.720946154s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.902279129 (246h25m4.721515153s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903290060 (246h25m4.722526084s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903897372 (246h25m4.723133396s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.904735619 (246h25m4.723971643s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.905349224 (246h25m4.724585248s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:19.706338743 (246h25m6.525574767s) [upload=236583]: Next returned nil segment Apr 17 22:11:19.706356788 (246h25m6.525592812s) [upload=236583]: Waiting for error group managing segments... Apr 17 22:11:19.994408492 (246h25m6.813644527s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:19.995247402 (246h25m6.814483426s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.782393945 (246h25m7.601629969s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.929683807 (246h25m7.748919831s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.038934967 (246h25m7.858170991s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.351460856 (246h25m8.17069688s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.445533741 (246h25m8.264769765s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.481223641 (246h25m8.300459665s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.603836618 (246h25m8.423072642s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.713343271 (246h25m8.532579295s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.729259759 (246h25m8.548495783s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.107132825 (246h25m8.926368849s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189431728 (246h25m9.008667762s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189455404 (246h25m9.008691428s) [seg_pos={0 0}, upload=236583]: Satisfied cohort requirements after 13 pieces Apr 17 22:11:22.189633718 (246h25m9.008869752s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189642284 (246h25m9.008878308s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189645831 (246h25m9.008881865s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189656893 (246h25m9.008892937s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189838644 (246h25m9.009074668s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189844715 (246h25m9.009080739s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189861708 (246h25m9.009097742s) [seg_pos={0 0}, upload=236583]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:11:22.189948866 (246h25m9.00918489s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:11:22.189954657 (246h25m9.009190691s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:11:22.483797551 (246h25m9.303033585s) [upload=236583]: Done uploading segments... Apr 17 22:11:22.483827999 (246h25m9.303064023s) [upload=236583]: Upload finished. err: <nil> Duration: 8.443020901s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...X.......................................XXXXX............] duration: 17.845945745s trace: https://jaeger.dev.storj.io/trace/1999777fac8036ca FAIL (2026-04-17 22:16:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.84596329s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [................X............X.XX...X.......XXXX.X.XX.XX.XXX] duration: 5.001594719s trace: https://jaeger.dev.storj.io/trace/22dd7253e8525caf FAIL (2026-04-17 22:16:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001616271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 8.353778567s trace: https://jaeger.dev.storj.io/trace/5cc52839840193df FAIL (2026-04-17 22:16:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:11:17.642192354 (246h25m4.461428378s) [upload=236583]: Starting upload Apr 17 22:11:17.642264764 (246h25m4.461500788s) [upload=236583]: Done starting upload Apr 17 22:11:17.642342734 (246h25m4.461578758s) [upload=236583]: Uploading segments... Apr 17 22:11:17.642525718 (246h25m4.461761742s) [upload=236583]: Got next segment. Inline: false Apr 17 22:11:17.642558962 (246h25m4.461794986s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:11:17.642563912 (246h25m4.461799936s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:11:17.882374413 (246h25m4.701610437s) [seg_pos={0 0}, upload=236583]: Begin upload segment... Apr 17 22:11:17.882438487 (246h25m4.701674511s) [seg_pos={0 0}, upload=236583]: Done begin upload segment. Apr 17 22:11:17.885163417 (246h25m4.704399471s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.890822067 (246h25m4.710058101s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.891352060 (246h25m4.710588084s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.893193865 (246h25m4.712429889s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895445962 (246h25m4.714681986s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895518891 (246h25m4.714754915s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.897292695 (246h25m4.716528719s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.899077650 (246h25m4.718313674s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900145952 (246h25m4.719381976s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900530905 (246h25m4.719766929s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.901710130 (246h25m4.720946154s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.902279129 (246h25m4.721515153s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903290060 (246h25m4.722526084s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903897372 (246h25m4.723133396s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.904735619 (246h25m4.723971643s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.905349224 (246h25m4.724585248s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:19.706338743 (246h25m6.525574767s) [upload=236583]: Next returned nil segment Apr 17 22:11:19.706356788 (246h25m6.525592812s) [upload=236583]: Waiting for error group managing segments... Apr 17 22:11:19.994408492 (246h25m6.813644527s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:19.995247402 (246h25m6.814483426s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.782393945 (246h25m7.601629969s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.929683807 (246h25m7.748919831s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.038934967 (246h25m7.858170991s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.351460856 (246h25m8.17069688s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.445533741 (246h25m8.264769765s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.481223641 (246h25m8.300459665s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.603836618 (246h25m8.423072642s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.713343271 (246h25m8.532579295s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.729259759 (246h25m8.548495783s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.107132825 (246h25m8.926368849s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189431728 (246h25m9.008667762s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189455404 (246h25m9.008691428s) [seg_pos={0 0}, upload=236583]: Satisfied cohort requirements after 13 pieces Apr 17 22:11:22.189633718 (246h25m9.008869752s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189642284 (246h25m9.008878308s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189645831 (246h25m9.008881865s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189656893 (246h25m9.008892937s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189838644 (246h25m9.009074668s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189844715 (246h25m9.009080739s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189861708 (246h25m9.009097742s) [seg_pos={0 0}, upload=236583]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:11:22.189948866 (246h25m9.00918489s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:11:22.189954657 (246h25m9.009190691s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:11:22.483797551 (246h25m9.303033585s) [upload=236583]: Done uploading segments... Apr 17 22:11:22.483827999 (246h25m9.303064023s) [upload=236583]: Upload finished. err: <nil> Duration: 8.353813134s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.................X............X.XX...X.......XXXX.X.XX.XX.XX] duration: 5.001444809s trace: https://jaeger.dev.storj.io/trace/5ea1d80e32a983ce FAIL (2026-04-17 22:15:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001463264s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....X.......................................XXXXX...........] duration: 29.533619715s trace: https://jaeger.dev.storj.io/trace/1999777fac8036ca FAIL (2026-04-17 22:15:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.533675773s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [..................X............X.XX...X.......XXXX.X.XX.XX.X] duration: 5.000764054s trace: https://jaeger.dev.storj.io/trace/70bf892bd1fc5bbb FAIL (2026-04-17 22:14:57 UTC) orbiter warning: context deadline exceeded Duration: 5.000792039s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 7.969582078s trace: https://jaeger.dev.storj.io/trace/5cc52839840193df FAIL (2026-04-17 22:14:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:11:17.642192354 (246h25m4.461428378s) [upload=236583]: Starting upload Apr 17 22:11:17.642264764 (246h25m4.461500788s) [upload=236583]: Done starting upload Apr 17 22:11:17.642342734 (246h25m4.461578758s) [upload=236583]: Uploading segments... Apr 17 22:11:17.642525718 (246h25m4.461761742s) [upload=236583]: Got next segment. Inline: false Apr 17 22:11:17.642558962 (246h25m4.461794986s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:11:17.642563912 (246h25m4.461799936s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:11:17.882374413 (246h25m4.701610437s) [seg_pos={0 0}, upload=236583]: Begin upload segment... Apr 17 22:11:17.882438487 (246h25m4.701674511s) [seg_pos={0 0}, upload=236583]: Done begin upload segment. Apr 17 22:11:17.885163417 (246h25m4.704399471s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.890822067 (246h25m4.710058101s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.891352060 (246h25m4.710588084s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.893193865 (246h25m4.712429889s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895445962 (246h25m4.714681986s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895518891 (246h25m4.714754915s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.897292695 (246h25m4.716528719s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.899077650 (246h25m4.718313674s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900145952 (246h25m4.719381976s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900530905 (246h25m4.719766929s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.901710130 (246h25m4.720946154s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.902279129 (246h25m4.721515153s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903290060 (246h25m4.722526084s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903897372 (246h25m4.723133396s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.904735619 (246h25m4.723971643s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.905349224 (246h25m4.724585248s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:19.706338743 (246h25m6.525574767s) [upload=236583]: Next returned nil segment Apr 17 22:11:19.706356788 (246h25m6.525592812s) [upload=236583]: Waiting for error group managing segments... Apr 17 22:11:19.994408492 (246h25m6.813644527s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:19.995247402 (246h25m6.814483426s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.782393945 (246h25m7.601629969s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.929683807 (246h25m7.748919831s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.038934967 (246h25m7.858170991s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.351460856 (246h25m8.17069688s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.445533741 (246h25m8.264769765s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.481223641 (246h25m8.300459665s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.603836618 (246h25m8.423072642s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.713343271 (246h25m8.532579295s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.729259759 (246h25m8.548495783s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.107132825 (246h25m8.926368849s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189431728 (246h25m9.008667762s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189455404 (246h25m9.008691428s) [seg_pos={0 0}, upload=236583]: Satisfied cohort requirements after 13 pieces Apr 17 22:11:22.189633718 (246h25m9.008869752s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189642284 (246h25m9.008878308s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189645831 (246h25m9.008881865s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189656893 (246h25m9.008892937s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189838644 (246h25m9.009074668s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189844715 (246h25m9.009080739s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189861708 (246h25m9.009097742s) [seg_pos={0 0}, upload=236583]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:11:22.189948866 (246h25m9.00918489s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:11:22.189954657 (246h25m9.009190691s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:11:22.483797551 (246h25m9.303033585s) [upload=236583]: Done uploading segments... Apr 17 22:11:22.483827999 (246h25m9.303064023s) [upload=236583]: Upload finished. err: <nil> Duration: 7.969618577s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....X.......................................XXXXX..........] duration: 5.774939765s trace: https://jaeger.dev.storj.io/trace/1999777fac8036ca FAIL (2026-04-17 22:14:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.774965386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...................X............X.XX...X.......XXXX.X.XX.XX.] duration: 3.457499764s trace: https://jaeger.dev.storj.io/trace/570672b8721b2396 FAIL (2026-04-17 22:14:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.457526987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 8.1125456s trace: https://jaeger.dev.storj.io/trace/5cc52839840193df FAIL (2026-04-17 22:13:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:11:17.642192354 (246h25m4.461428378s) [upload=236583]: Starting upload Apr 17 22:11:17.642264764 (246h25m4.461500788s) [upload=236583]: Done starting upload Apr 17 22:11:17.642342734 (246h25m4.461578758s) [upload=236583]: Uploading segments... Apr 17 22:11:17.642525718 (246h25m4.461761742s) [upload=236583]: Got next segment. Inline: false Apr 17 22:11:17.642558962 (246h25m4.461794986s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:11:17.642563912 (246h25m4.461799936s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:11:17.882374413 (246h25m4.701610437s) [seg_pos={0 0}, upload=236583]: Begin upload segment... Apr 17 22:11:17.882438487 (246h25m4.701674511s) [seg_pos={0 0}, upload=236583]: Done begin upload segment. Apr 17 22:11:17.885163417 (246h25m4.704399471s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.890822067 (246h25m4.710058101s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.891352060 (246h25m4.710588084s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.893193865 (246h25m4.712429889s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895445962 (246h25m4.714681986s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895518891 (246h25m4.714754915s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.897292695 (246h25m4.716528719s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.899077650 (246h25m4.718313674s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900145952 (246h25m4.719381976s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900530905 (246h25m4.719766929s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.901710130 (246h25m4.720946154s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.902279129 (246h25m4.721515153s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903290060 (246h25m4.722526084s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903897372 (246h25m4.723133396s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.904735619 (246h25m4.723971643s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.905349224 (246h25m4.724585248s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:19.706338743 (246h25m6.525574767s) [upload=236583]: Next returned nil segment Apr 17 22:11:19.706356788 (246h25m6.525592812s) [upload=236583]: Waiting for error group managing segments... Apr 17 22:11:19.994408492 (246h25m6.813644527s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:19.995247402 (246h25m6.814483426s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.782393945 (246h25m7.601629969s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.929683807 (246h25m7.748919831s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.038934967 (246h25m7.858170991s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.351460856 (246h25m8.17069688s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.445533741 (246h25m8.264769765s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.481223641 (246h25m8.300459665s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.603836618 (246h25m8.423072642s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.713343271 (246h25m8.532579295s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.729259759 (246h25m8.548495783s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.107132825 (246h25m8.926368849s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189431728 (246h25m9.008667762s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189455404 (246h25m9.008691428s) [seg_pos={0 0}, upload=236583]: Satisfied cohort requirements after 13 pieces Apr 17 22:11:22.189633718 (246h25m9.008869752s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189642284 (246h25m9.008878308s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189645831 (246h25m9.008881865s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189656893 (246h25m9.008892937s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189838644 (246h25m9.009074668s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189844715 (246h25m9.009080739s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189861708 (246h25m9.009097742s) [seg_pos={0 0}, upload=236583]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:11:22.189948866 (246h25m9.00918489s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:11:22.189954657 (246h25m9.009190691s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:11:22.483797551 (246h25m9.303033585s) [upload=236583]: Done uploading segments... Apr 17 22:11:22.483827999 (246h25m9.303064023s) [upload=236583]: Upload finished. err: <nil> Duration: 8.11256618s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [....................X............X.XX...X.......XXXX.X.XX.XX] duration: 5.00104594s trace: https://jaeger.dev.storj.io/trace/570672b8721b2396 FAIL (2026-04-17 22:13:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001069556s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......X.......................................XXXXX.........] duration: 9.643217971s trace: https://jaeger.dev.storj.io/trace/1999777fac8036ca FAIL (2026-04-17 22:13:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.643240755s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.....................X............X.XX...X.......XXXX.X.XX.X] duration: 5.001013961s trace: https://jaeger.dev.storj.io/trace/73d58f6f18d44c1 FAIL (2026-04-17 22:12:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00103416s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 7.659229804s trace: https://jaeger.dev.storj.io/trace/5cc52839840193df FAIL (2026-04-17 22:12:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:11:17.642192354 (246h25m4.461428378s) [upload=236583]: Starting upload Apr 17 22:11:17.642264764 (246h25m4.461500788s) [upload=236583]: Done starting upload Apr 17 22:11:17.642342734 (246h25m4.461578758s) [upload=236583]: Uploading segments... Apr 17 22:11:17.642525718 (246h25m4.461761742s) [upload=236583]: Got next segment. Inline: false Apr 17 22:11:17.642558962 (246h25m4.461794986s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:11:17.642563912 (246h25m4.461799936s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:11:17.882374413 (246h25m4.701610437s) [seg_pos={0 0}, upload=236583]: Begin upload segment... Apr 17 22:11:17.882438487 (246h25m4.701674511s) [seg_pos={0 0}, upload=236583]: Done begin upload segment. Apr 17 22:11:17.885163417 (246h25m4.704399471s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.890822067 (246h25m4.710058101s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.891352060 (246h25m4.710588084s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.893193865 (246h25m4.712429889s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895445962 (246h25m4.714681986s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895518891 (246h25m4.714754915s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.897292695 (246h25m4.716528719s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.899077650 (246h25m4.718313674s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900145952 (246h25m4.719381976s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900530905 (246h25m4.719766929s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.901710130 (246h25m4.720946154s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.902279129 (246h25m4.721515153s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903290060 (246h25m4.722526084s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903897372 (246h25m4.723133396s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.904735619 (246h25m4.723971643s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.905349224 (246h25m4.724585248s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:19.706338743 (246h25m6.525574767s) [upload=236583]: Next returned nil segment Apr 17 22:11:19.706356788 (246h25m6.525592812s) [upload=236583]: Waiting for error group managing segments... Apr 17 22:11:19.994408492 (246h25m6.813644527s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:19.995247402 (246h25m6.814483426s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.782393945 (246h25m7.601629969s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.929683807 (246h25m7.748919831s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.038934967 (246h25m7.858170991s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.351460856 (246h25m8.17069688s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.445533741 (246h25m8.264769765s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.481223641 (246h25m8.300459665s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.603836618 (246h25m8.423072642s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.713343271 (246h25m8.532579295s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.729259759 (246h25m8.548495783s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.107132825 (246h25m8.926368849s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189431728 (246h25m9.008667762s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189455404 (246h25m9.008691428s) [seg_pos={0 0}, upload=236583]: Satisfied cohort requirements after 13 pieces Apr 17 22:11:22.189633718 (246h25m9.008869752s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189642284 (246h25m9.008878308s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189645831 (246h25m9.008881865s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189656893 (246h25m9.008892937s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189838644 (246h25m9.009074668s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189844715 (246h25m9.009080739s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189861708 (246h25m9.009097742s) [seg_pos={0 0}, upload=236583]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:11:22.189948866 (246h25m9.00918489s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:11:22.189954657 (246h25m9.009190691s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:11:22.483797551 (246h25m9.303033585s) [upload=236583]: Done uploading segments... Apr 17 22:11:22.483827999 (246h25m9.303064023s) [upload=236583]: Upload finished. err: <nil> Duration: 7.659260634s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.......X.......................................XXXXX........] duration: 16.135152313s trace: https://jaeger.dev.storj.io/trace/1999777fac8036ca FAIL (2026-04-17 22:12:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.135180305s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [......................X............X.XX...X.......XXXX.X.XX.] duration: 4.384221101s trace: https://jaeger.dev.storj.io/trace/39986dc68e35954 FAIL (2026-04-17 22:12:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.384249234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 9.257451028s trace: https://jaeger.dev.storj.io/trace/5cc52839840193df FAIL (2026-04-17 22:12:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:11:17.642192354 (246h25m4.461428378s) [upload=236583]: Starting upload Apr 17 22:11:17.642264764 (246h25m4.461500788s) [upload=236583]: Done starting upload Apr 17 22:11:17.642342734 (246h25m4.461578758s) [upload=236583]: Uploading segments... Apr 17 22:11:17.642525718 (246h25m4.461761742s) [upload=236583]: Got next segment. Inline: false Apr 17 22:11:17.642558962 (246h25m4.461794986s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:11:17.642563912 (246h25m4.461799936s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:11:17.882374413 (246h25m4.701610437s) [seg_pos={0 0}, upload=236583]: Begin upload segment... Apr 17 22:11:17.882438487 (246h25m4.701674511s) [seg_pos={0 0}, upload=236583]: Done begin upload segment. Apr 17 22:11:17.885163417 (246h25m4.704399471s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.890822067 (246h25m4.710058101s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.891352060 (246h25m4.710588084s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.893193865 (246h25m4.712429889s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895445962 (246h25m4.714681986s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895518891 (246h25m4.714754915s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.897292695 (246h25m4.716528719s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.899077650 (246h25m4.718313674s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900145952 (246h25m4.719381976s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900530905 (246h25m4.719766929s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.901710130 (246h25m4.720946154s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.902279129 (246h25m4.721515153s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903290060 (246h25m4.722526084s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903897372 (246h25m4.723133396s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.904735619 (246h25m4.723971643s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.905349224 (246h25m4.724585248s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:19.706338743 (246h25m6.525574767s) [upload=236583]: Next returned nil segment Apr 17 22:11:19.706356788 (246h25m6.525592812s) [upload=236583]: Waiting for error group managing segments... Apr 17 22:11:19.994408492 (246h25m6.813644527s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:19.995247402 (246h25m6.814483426s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.782393945 (246h25m7.601629969s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.929683807 (246h25m7.748919831s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.038934967 (246h25m7.858170991s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.351460856 (246h25m8.17069688s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.445533741 (246h25m8.264769765s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.481223641 (246h25m8.300459665s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.603836618 (246h25m8.423072642s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.713343271 (246h25m8.532579295s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.729259759 (246h25m8.548495783s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.107132825 (246h25m8.926368849s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189431728 (246h25m9.008667762s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189455404 (246h25m9.008691428s) [seg_pos={0 0}, upload=236583]: Satisfied cohort requirements after 13 pieces Apr 17 22:11:22.189633718 (246h25m9.008869752s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189642284 (246h25m9.008878308s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189645831 (246h25m9.008881865s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189656893 (246h25m9.008892937s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189838644 (246h25m9.009074668s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189844715 (246h25m9.009080739s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189861708 (246h25m9.009097742s) [seg_pos={0 0}, upload=236583]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:11:22.189948866 (246h25m9.00918489s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:11:22.189954657 (246h25m9.009190691s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:11:22.483797551 (246h25m9.303033585s) [upload=236583]: Done uploading segments... Apr 17 22:11:22.483827999 (246h25m9.303064023s) [upload=236583]: Upload finished. err: <nil> Duration: 9.257476257s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.......................X............X.XX...X.......XXXX.X.XX] duration: 5.001558869s trace: https://jaeger.dev.storj.io/trace/39986dc68e35954 FAIL (2026-04-17 22:11:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001577695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002849942s trace: https://jaeger.dev.storj.io/trace/5cc52839840193df FAIL (2026-04-17 22:11:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 17 22:11:17.642192354 (246h25m4.461428378s) [upload=236583]: Starting upload Apr 17 22:11:17.642264764 (246h25m4.461500788s) [upload=236583]: Done starting upload Apr 17 22:11:17.642342734 (246h25m4.461578758s) [upload=236583]: Uploading segments... Apr 17 22:11:17.642525718 (246h25m4.461761742s) [upload=236583]: Got next segment. Inline: false Apr 17 22:11:17.642558962 (246h25m4.461794986s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 17 22:11:17.642563912 (246h25m4.461799936s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 17 22:11:17.882374413 (246h25m4.701610437s) [seg_pos={0 0}, upload=236583]: Begin upload segment... Apr 17 22:11:17.882438487 (246h25m4.701674511s) [seg_pos={0 0}, upload=236583]: Done begin upload segment. Apr 17 22:11:17.885163417 (246h25m4.704399471s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.890822067 (246h25m4.710058101s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.891352060 (246h25m4.710588084s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.893193865 (246h25m4.712429889s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895445962 (246h25m4.714681986s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.895518891 (246h25m4.714754915s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.897292695 (246h25m4.716528719s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.899077650 (246h25m4.718313674s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900145952 (246h25m4.719381976s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.900530905 (246h25m4.719766929s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.901710130 (246h25m4.720946154s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.902279129 (246h25m4.721515153s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903290060 (246h25m4.722526084s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.903897372 (246h25m4.723133396s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.904735619 (246h25m4.723971643s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:17.905349224 (246h25m4.724585248s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Uploading piece... Apr 17 22:11:19.706338743 (246h25m6.525574767s) [upload=236583]: Next returned nil segment Apr 17 22:11:19.706356788 (246h25m6.525592812s) [upload=236583]: Waiting for error group managing segments... Apr 17 22:11:19.994408492 (246h25m6.813644527s) [noise=true , address=77.90.31.18:4810 , piece_id=N7XFDY27NJTWOYMIS5KEGBSAUTJ5BXQXNTWD367YSRRNAB5WLNOA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:19.995247402 (246h25m6.814483426s) [noise=true , address=77.90.31.10:4521 , piece_id=OD6737KJZK3DYXTY7KVHT7YM4EZJ5G3PLPM24OV5K6QSVT6LQAZA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.782393945 (246h25m7.601629969s) [noise=true , address=145.79.227.14:4837 , piece_id=ZOMRP2QCEPGDJ7NNE2QROG5KH73RTJ2LM2DS763TKBUULAJ2G37Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:20.929683807 (246h25m7.748919831s) [noise=true , address=145.79.227.2:4116 , piece_id=QWMDOBFXL3C3MC4UAH32YZYTEGG2YYDXYMPMDJZZYHWEQLZIUOTA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.038934967 (246h25m7.858170991s) [noise=true , address=145.79.227.3:4150 , piece_id=NLEB5GCUK4OXMZ43GG33SC7PKSHNXRFN4N7OCT37VIJZLCZ7SRXA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.351460856 (246h25m8.17069688s) [noise=true , address=145.79.227.8:4331 , piece_id=VO4NPYQ3HAC35SP5ZTKZ4RIW4WUUHUBZMF5VZWJOVGWFQ36ETC4Q, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.445533741 (246h25m8.264769765s) [noise=true , address=145.79.227.12:4769 , piece_id=D7OX66PSEP4A2H5A64MA2MVSPT3H7BVQAU3PPNPXA2HM4DC4UEHA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.481223641 (246h25m8.300459665s) [noise=true , address=45.91.116.9:4296 , piece_id=SHDLZOWNC7VLN63OLW24FKRJTWDGOOQA6OBBXHIILSZYPHHZKHDQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.603836618 (246h25m8.423072642s) [noise=true , address=45.91.116.12:4226 , piece_id=BYP6OEG4PM5NBAJMKXVMNDJAIBHZD237OSTHIHPQLQZREHWVDZFQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.713343271 (246h25m8.532579295s) [noise=true , address=45.91.116.7:4024 , piece_id=5X7NBLIFKJBYSBV4IPBPID4OON3IAVWOIKKFLJKSBFC6ZXCU6OWA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:21.729259759 (246h25m8.548495783s) [noise=true , address=145.79.227.26:4658 , piece_id=MBX7N2AP7RC4EHJVRLTC53TUHIGVLFH3SEN3FPRJVAGT43J5NUTQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.107132825 (246h25m8.926368849s) [noise=true , address=145.79.227.31:4801 , piece_id=B6LP62UZ3SHLTKQK54ZMVE3RPSRHYBPLRMGLQX6MME54ZT7NENJQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189431728 (246h25m9.008667762s) [noise=true , address=145.79.227.23:4551 , piece_id=JPWBC5NCZSXSHXO757XF2SKIZZQUCI3RM5IMVTROOQHL5FVW5E2A, seg_pos={0 0}, upload=236583]: Done uploading piece. err: <nil> Apr 17 22:11:22.189455404 (246h25m9.008691428s) [seg_pos={0 0}, upload=236583]: Satisfied cohort requirements after 13 pieces Apr 17 22:11:22.189633718 (246h25m9.008869752s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189642284 (246h25m9.008878308s) [noise=true , address=145.79.227.35:4486 , piece_id=P26B5NG7QTTTRJ4RM4NQWRQ5HYTINX6SQV6NRUQKX7ITW4EAIFOQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189645831 (246h25m9.008881865s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189656893 (246h25m9.008892937s) [noise=true , address=145.79.227.5:4226 , piece_id=EII3WG5HI7MAUGCRFBL2CIFL5R3QA2M4RHZXZJLZPGSFVVV3VSYQ, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189838644 (246h25m9.009074668s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Apr 17 22:11:22.189844715 (246h25m9.009080739s) [noise=true , address=145.79.227.43:4159 , piece_id=TJQOYPQX7N3QLTA6H5J7E6KCRQUZYAHLCNIO4U5K5YGLRRZSTHCA, seg_pos={0 0}, upload=236583]: Long tail cancellation due to optimal threshold Apr 17 22:11:22.189861708 (246h25m9.009097742s) [seg_pos={0 0}, upload=236583]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 17 22:11:22.189948866 (246h25m9.00918489s) [upload=236583]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 17 22:11:22.189954657 (246h25m9.009190691s) [upload=236583]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 17 22:11:22.483797551 (246h25m9.303033585s) [upload=236583]: Done uploading segments... Apr 17 22:11:22.483827999 (246h25m9.303064023s) [upload=236583]: Upload finished. err: <nil> Duration: 30.002871122s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........X.......................................XXXXX.......] duration: 9.451859933s trace: https://jaeger.dev.storj.io/trace/1999777fac8036ca FAIL (2026-04-17 22:11:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.451891824s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........X.......................................XXXXX......] duration: 14.432381467s trace: https://jaeger.dev.storj.io/trace/1999777fac8036ca FAIL (2026-04-17 22:11:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.432403339s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..........X.......................................XXXXX.....] duration: 11.957057739s trace: https://jaeger.dev.storj.io/trace/1999777fac8036ca FAIL (2026-04-17 22:11:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.957086926s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [........................X............X.XX...X.......XXXX.X.X] duration: 5.001570941s trace: https://jaeger.dev.storj.io/trace/69293eea35491224 FAIL (2026-04-17 22:10:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001583885s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...........X.......................................XXXXX....] duration: 21.437418298s trace: https://jaeger.dev.storj.io/trace/1999777fac8036ca FAIL (2026-04-17 22:10:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.437491348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.........................X............X.XX...X.......XXXX.X.] duration: 3.989013394s trace: https://jaeger.dev.storj.io/trace/647533b91507d3c9 FAIL (2026-04-17 22:10:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.989040557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [............X.......................................XXXXX...] duration: 10.118843564s trace: https://jaeger.dev.storj.io/trace/1999777fac8036ca FAIL (2026-04-17 22:09:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.118863603s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)