Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 404805 reports (Page 282 of 8097)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X..X.X..........X...XX...............XX.........X.......X.X] duration: 30.003283986s trace: https://jaeger.dev.storj.io/trace/2a0b9748c78e5bb5 FAIL (2026-01-05 12:53:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 5 12:53:22.980192976 (741h24m22.851276021s) [upload=648704]: Starting upload Jan 5 12:53:22.980363470 (741h24m22.851446516s) [upload=648704]: Done starting upload Jan 5 12:53:22.980425148 (741h24m22.851508183s) [upload=648704]: Uploading segments... Jan 5 12:53:22.980598318 (741h24m22.851681353s) [upload=648704]: Got next segment. Inline: false Jan 5 12:53:22.980667460 (741h24m22.851750505s) [upload=648704]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 5 12:53:22.980673791 (741h24m22.851756827s) [upload=648704]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 5 12:53:23.227527179 (741h24m23.098610224s) [seg_pos={0 0}, upload=648704]: Begin upload segment... Jan 5 12:53:23.227602983 (741h24m23.098686028s) [seg_pos={0 0}, upload=648704]: Done begin upload segment. Jan 5 12:53:23.227669500 (741h24m23.098752545s) [noise=true , address=145.79.227.48:4302 , piece_id=DDADMJC47364YJQ6ZLD7RC5OBZZUYB6TEJOWYEZXJHHQNMRUQ46Q, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.227718123 (741h24m23.098801158s) [noise=true , address=45.91.116.6:4974 , piece_id=Y6RMRIO5HJMVUG5YFPBAJ23PXB6XI4UDE4E2WOHPLVSU276NUQOQ, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.229385037 (741h24m23.100468072s) [noise=true , address=45.91.116.14:4191 , piece_id=6XSOJFJNRRAYDV7YOQVNP3243LAE6HEBGPBDRAL7KHCKDVHQ5LKA, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.229468315 (741h24m23.10055136s) [noise=true , address=145.79.227.26:4653 , piece_id=VQFN24T2QYUFYCWU5GJKAUBGZD3676VU2WBUB3CCSCNJSPTFOGYA, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.229978537 (741h24m23.101061582s) [noise=true , address=77.90.31.15:4700 , piece_id=NLREKLP3HO67I4NBP7RSVGY4LUO4YQ3B6Y4LH4RYQCGO47ZWPRSA, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.230422301 (741h24m23.101505336s) [noise=true , address=145.79.227.39:4015 , piece_id=RBL63OQPTQFY5T2NQFYYX6D2AH335VL4SOVGSZM6IPVNXFP6DJ5A, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.230558451 (741h24m23.101641496s) [noise=true , address=45.91.116.8:4045 , piece_id=VGFSXFTBQ53RLGAMBQMCVLR3ABGG7ZUAO4HIXPYTBSOMZRCR2OVA, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.231057522 (741h24m23.102140557s) [noise=true , address=145.79.227.47:4266 , piece_id=HWZB2L2YSHUU2AI3OWSG3JFYW22W5OHLSSYZR2C4SVMY2B7WSO6A, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.231157632 (741h24m23.102240677s) [noise=true , address=145.79.227.29:4943 , piece_id=QTOB56BQSM3NDQVTG6XDZTBYXGLIWX3PIJOBIIU2QNIWBBJKJH3Q, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.232627271 (741h24m23.103710316s) [noise=true , address=145.79.227.15:4873 , piece_id=J2L7MHYR7U55UA5JEV22P5QI6JFFR3ISEVZF6ON5PEWIWX2I74ZA, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.233229537 (741h24m23.104312582s) [noise=true , address=77.90.31.16:4735 , piece_id=HRV56JZJZOW2Y6ZO3RF4L3HTWCAYCCDI67FDCALOEVK62I5LBW3Q, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.233904652 (741h24m23.104987687s) [noise=true , address=45.91.116.5:4297 , piece_id=7YPTRZK55S6D44BVJWBVEXPSMX3UADWFVDFSSO3HHXGHEWJBXZYQ, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.234542497 (741h24m23.105625533s) [noise=true , address=145.79.227.27:4691 , piece_id=RPTX7PHYOYCIJWDOAHNH554Q7DLITDUWTG7US3XYD6ZJKH4I4Z7A, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.235169381 (741h24m23.106252426s) [noise=true , address=145.79.227.51:4411 , piece_id=TBAKPH5YZJDMBCDWAPY23Q33WXX5FMOV7Y3LBB44MFWSKXZGBSNA, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.237121699 (741h24m23.108204744s) [noise=true , address=145.79.227.35:4483 , piece_id=XKKC63QZJYSRXSZI7IYEP4GO225TWKZVBRZ53N7PRU3NZ2OIOMQA, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:23.239140593 (741h24m23.110223638s) [noise=true , address=145.79.227.3:4151 , piece_id=E2CTDKKY4POBUJ4JKZTDUD7SGBJB2MBENZLPUYXAUJNEIQK3B5NQ, seg_pos={0 0}, upload=648704]: Uploading piece... Jan 5 12:53:24.941255281 (741h24m24.812338316s) [upload=648704]: Next returned nil segment Jan 5 12:53:24.941307451 (741h24m24.812390516s) [upload=648704]: Waiting for error group managing segments... Jan 5 12:53:25.228913327 (741h24m25.099996372s) [noise=true , address=145.79.227.26:4653 , piece_id=VQFN24T2QYUFYCWU5GJKAUBGZD3676VU2WBUB3CCSCNJSPTFOGYA, seg_pos={0 0}, upload=648704]: Done uploading piece. err: <nil> Jan 5 12:53:25.229717338 (741h24m25.100800373s) [noise=true , address=145.79.227.39:4015 , piece_id=RBL63OQPTQFY5T2NQFYYX6D2AH335VL4SOVGSZM6IPVNXFP6DJ5A, seg_pos={0 0}, upload=648704]: Done uploading piece. err: <nil> Jan 5 12:53:25.236142943 (741h24m25.107225988s) [noise=true , address=145.79.227.48:4302 , piece_id=DDADMJC47364YJQ6ZLD7RC5OBZZUYB6TEJOWYEZXJHHQNMRUQ46Q, seg_pos={0 0}, upload=648704]: Done uploading piece. err: <nil> Jan 5 12:53:25.263276580 (741h24m25.134359615s) [noise=true , address=145.79.227.3:4151 , piece_id=E2CTDKKY4POBUJ4JKZTDUD7SGBJB2MBENZLPUYXAUJNEIQK3B5NQ, seg_pos={0 0}, upload=648704]: Done uploading piece. err: <nil> Jan 5 12:53:25.309531566 (741h24m25.180614601s) [noise=true , address=145.79.227.15:4873 , piece_id=J2L7MHYR7U55UA5JEV22P5QI6JFFR3ISEVZF6ON5PEWIWX2I74ZA, seg_pos={0 0}, upload=648704]: Done uploading piece. err: <nil> Jan 5 12:53:25.314730737 (741h24m25.185813782s) [noise=true , address=145.79.227.51:4411 , piece_id=TBAKPH5YZJDMBCDWAPY23Q33WXX5FMOV7Y3LBB44MFWSKXZGBSNA, seg_pos={0 0}, upload=648704]: Done uploading piece. err: <nil> Jan 5 12:53:25.317508796 (741h24m25.188591841s) [noise=true , address=145.79.227.47:4266 , piece_id=HWZB2L2YSHUU2AI3OWSG3JFYW22W5OHLSSYZR2C4SVMY2B7WSO6A, seg_pos={0 0}, upload=648704]: Done uploading piece. err: <nil> Jan 5 12:53:25.401644745 (741h24m25.27272778s) [noise=true , address=145.79.227.27:4691 , piece_id=RPTX7PHYOYCIJWDOAHNH554Q7DLITDUWTG7US3XYD6ZJKH4I4Z7A, seg_pos={0 0}, upload=648704]: Done uploading piece. err: <nil> Jan 5 12:53:25.552606449 (741h24m25.423689494s) [noise=true , address=77.90.31.15:4700 , piece_id=NLREKLP3HO67I4NBP7RSVGY4LUO4YQ3B6Y4LH4RYQCGO47ZWPRSA, seg_pos={0 0}, upload=648704]: Done uploading piece. err: <nil> Jan 5 12:53:25.837399319 (741h24m25.708482384s) [noise=true , address=45.91.116.14:4191 , piece_id=6XSOJFJNRRAYDV7YOQVNP3243LAE6HEBGPBDRAL7KHCKDVHQ5LKA, seg_pos={0 0}, upload=648704]: Done uploading piece. err: <nil> Jan 5 12:53:26.136622876 (741h24m26.007705921s) [noise=true , address=45.91.116.8:4045 , piece_id=VGFSXFTBQ53RLGAMBQMCVLR3ABGG7ZUAO4HIXPYTBSOMZRCR2OVA, seg_pos={0 0}, upload=648704]: Done uploading piece. err: <nil> Jan 5 12:53:26.536899185 (741h24m26.40798223s) [noise=true , address=145.79.227.35:4483 , piece_id=XKKC63QZJYSRXSZI7IYEP4GO225TWKZVBRZ53N7PRU3NZ2OIOMQA, seg_pos={0 0}, upload=648704]: Done uploading piece. err: <nil> Jan 5 12:53:26.567651160 (741h24m26.438734206s) [noise=true , address=145.79.227.29:4943 , piece_id=QTOB56BQSM3NDQVTG6XDZTBYXGLIWX3PIJOBIIU2QNIWBBJKJH3Q, seg_pos={0 0}, upload=648704]: Done uploading piece. err: <nil> Jan 5 12:53:26.567678523 (741h24m26.438761568s) [seg_pos={0 0}, upload=648704]: Segment reached optimal threshold of 13 pieces. Jan 5 12:53:26.567946924 (741h24m26.439029989s) [noise=true , address=77.90.31.16:4735 , piece_id=HRV56JZJZOW2Y6ZO3RF4L3HTWCAYCCDI67FDCALOEVK62I5LBW3Q, seg_pos={0 0}, upload=648704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jan 5 12:53:26.567956783 (741h24m26.439039828s) [noise=true , address=77.90.31.16:4735 , piece_id=HRV56JZJZOW2Y6ZO3RF4L3HTWCAYCCDI67FDCALOEVK62I5LBW3Q, seg_pos={0 0}, upload=648704]: Long tail cancellation due to optimal threshold Jan 5 12:53:26.568276131 (741h24m26.439359196s) [noise=true , address=45.91.116.6:4974 , piece_id=Y6RMRIO5HJMVUG5YFPBAJ23PXB6XI4UDE4E2WOHPLVSU276NUQOQ, seg_pos={0 0}, upload=648704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Jan 5 12:53:26.568287221 (741h24m26.439370266s) [noise=true , address=45.91.116.6:4974 , piece_id=Y6RMRIO5HJMVUG5YFPBAJ23PXB6XI4UDE4E2WOHPLVSU276NUQOQ, seg_pos={0 0}, upload=648704]: Long tail cancellation due to optimal threshold Jan 5 12:53:26.568486321 (741h24m26.439569386s) [noise=true , address=45.91.116.5:4297 , piece_id=7YPTRZK55S6D44BVJWBVEXPSMX3UADWFVDFSSO3HHXGHEWJBXZYQ, seg_pos={0 0}, upload=648704]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 5 12:53:26.568495338 (741h24m26.439578383s) [noise=true , address=45.91.116.5:4297 , piece_id=7YPTRZK55S6D44BVJWBVEXPSMX3UADWFVDFSSO3HHXGHEWJBXZYQ, seg_pos={0 0}, upload=648704]: Long tail cancellation due to optimal threshold Jan 5 12:53:26.568516558 (741h24m26.439599594s) [seg_pos={0 0}, upload=648704]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 5 12:53:26.568603034 (741h24m26.439686089s) [upload=648704]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:53:26.568612161 (741h24m26.439695206s) [upload=648704]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:53:26.867393657 (741h24m26.738476692s) [upload=648704]: Done uploading segments... Jan 5 12:53:26.867412453 (741h24m26.738495488s) [upload=648704]: Upload finished. err: <nil> Duration: 30.003306669s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............X.............X..............................] duration: 5.903151946s trace: https://jaeger.dev.storj.io/trace/5f0936145b5399cd FAIL (2026-01-05 12:53:41 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.90318047s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...........X....X.X.....X.....X..X..X...X....XX.X...XXX.....] duration: 30.001601341s trace: https://jaeger.dev.storj.io/trace/16381dc260dc9d1a FAIL (2026-01-05 12:53:39 UTC) orbiter warning: context deadline exceeded Duration: 30.001627269s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX.XX.X....X....X..XX..X.XXX.X.XXX....X....XX.....XX.XXX...X] duration: 31.090397556s trace: https://jaeger.dev.storj.io/trace/28c185a3a8b490c4 FAIL (2026-01-05 12:53:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.225830251 (741h23m54.096913296s) [noise=true , address=145.79.227.7:4313 , piece_id=N6QI3KQXV5CDY6NQHHHBS7HLD24725E5EYN3FGWTKWSUFVNHB7JA, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.226305165 (741h23m54.09738822s) [noise=true , address=145.79.227.43:4177 , piece_id=PXINTEEBYFDYXSF2BTQ5ZLJDZ7IKIJOBBQPQUKQD26QXUT4RZ3UA, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.226551444 (741h23m54.097634489s) [noise=true , address=45.91.116.8:4064 , piece_id=FNB4SN5DOBWXG23BKHN5EOACLXI3NYGOM75Q5YTWGQDJYYVWPO4Q, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.226592623 (741h23m54.097675658s) [noise=true , address=64.49.13.169:4711 , piece_id=JRWRCUZZW25F7HEAGGPZQU3CG6QOF33S34ZR54WTD4L2N6S675MA, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.226877665 (741h23m54.09796074s) [noise=true , address=145.79.227.44:4215 , piece_id=GQG6VMUBGSLTUQDG5FVQTDUKZYPJ7QZ2MANUHZJQ2W7DZTNSXA5A, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.227046507 (741h23m54.098129552s) [noise=true , address=145.79.227.35:4495 , piece_id=UUNF3YMZ3OMOMNDFEZ245XHFJIDZXS5JP5AIKCDJRK7VW5FGYAKA, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.227186413 (741h23m54.098269458s) [noise=true , address=77.90.31.10:4542 , piece_id=LEU44EKI74PF6L62ELQVNKH6S3RLI2QWOZIZBMMCML76NTLWSJDQ, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.227472658 (741h23m54.098555703s) [noise=true , address=145.79.227.30:4777 , piece_id=KFLX343V4QY7UZIBHN25CZLXWGBRV724VNZSDDCAGWQ7NVOW36NQ, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.227747251 (741h23m54.098830296s) [noise=true , address=77.90.31.15:4716 , piece_id=FTUXZUD2VOFOBY42VU77O2KZ2Q5J6ETABI754OX4E5EVQUDH4ILQ, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.227763221 (741h23m54.098846266s) [noise=true , address=77.90.31.17:4798 , piece_id=MQD5EH4BKB6NYRRCOPDEU5EJ5H5TB5A7QYFI34BJX2PBLXVQ3RBQ, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.228919743 (741h23m54.100002798s) [noise=true , address=145.79.227.49:4359 , piece_id=ROW7SEA44EIQPIA4ES2Q5KQD2FQX2OBISSK44T563MBDJCVYIAVA, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.229482054 (741h23m54.100565089s) [noise=true , address=45.91.116.12:4230 , piece_id=TKXW6FBC4AB4CHGUBK4O67PCQKAERO637ZCRWRLNWS4J3B5HOJKA, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.229951398 (741h23m54.101034433s) [noise=true , address=45.91.116.14:4201 , piece_id=B4RBSL2C3K7LDTCJKMVIVABOFUVT3NQNDTJXDV4X6NRTQDUSTFTQ, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.231400597 (741h23m54.102483642s) [noise=true , address=145.79.227.40:4053 , piece_id=RVRFEMIOPASHB5NRRUWDE6T36L3NCR7T7GECNA3BLF76X3RLVI6A, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:54.231884599 (741h23m54.102967634s) [noise=true , address=145.79.227.34:4939 , piece_id=YPX3IEEBUFQVPLIZBIUSYLUPZ7PVH6JQCNOU2SZLIYZO2A6VXH4A, seg_pos={0 0}, upload=648698]: Uploading piece... Jan 5 12:52:55.638607797 (741h23m55.509690832s) [upload=648698]: Next returned nil segment Jan 5 12:52:55.638623967 (741h23m55.509707012s) [upload=648698]: Waiting for error group managing segments... Jan 5 12:52:56.079598271 (741h23m55.950681316s) [noise=true , address=145.79.227.7:4313 , piece_id=N6QI3KQXV5CDY6NQHHHBS7HLD24725E5EYN3FGWTKWSUFVNHB7JA, seg_pos={0 0}, upload=648698]: Done uploading piece. err: <nil> Jan 5 12:52:56.170780947 (741h23m56.041863982s) [noise=true , address=145.79.227.49:4359 , piece_id=ROW7SEA44EIQPIA4ES2Q5KQD2FQX2OBISSK44T563MBDJCVYIAVA, seg_pos={0 0}, upload=648698]: Done uploading piece. err: <nil> Jan 5 12:52:56.438850803 (741h23m56.309933838s) [noise=true , address=77.90.31.17:4798 , piece_id=MQD5EH4BKB6NYRRCOPDEU5EJ5H5TB5A7QYFI34BJX2PBLXVQ3RBQ, seg_pos={0 0}, upload=648698]: Done uploading piece. err: <nil> Jan 5 12:52:56.561348809 (741h23m56.432431854s) [noise=true , address=145.79.227.35:4495 , piece_id=UUNF3YMZ3OMOMNDFEZ245XHFJIDZXS5JP5AIKCDJRK7VW5FGYAKA, seg_pos={0 0}, upload=648698]: Done uploading piece. err: <nil> Jan 5 12:52:56.988099728 (741h23m56.859182763s) [noise=true , address=145.79.227.30:4777 , piece_id=KFLX343V4QY7UZIBHN25CZLXWGBRV724VNZSDDCAGWQ7NVOW36NQ, seg_pos={0 0}, upload=648698]: Done uploading piece. err: <nil> Jan 5 12:52:57.009014423 (741h23m56.880097468s) [noise=true , address=145.79.227.44:4215 , piece_id=GQG6VMUBGSLTUQDG5FVQTDUKZYPJ7QZ2MANUHZJQ2W7DZTNSXA5A, seg_pos={0 0}, upload=648698]: Done uploading piece. err: <nil> Jan 5 12:52:57.019838023 (741h23m56.890921068s) [noise=true , address=145.79.227.34:4939 , piece_id=YPX3IEEBUFQVPLIZBIUSYLUPZ7PVH6JQCNOU2SZLIYZO2A6VXH4A, seg_pos={0 0}, upload=648698]: Done uploading piece. err: <nil> Jan 5 12:52:57.208732787 (741h23m57.079815832s) [noise=true , address=145.79.227.33:4876 , piece_id=MAMVTCAOUMMYGIV4XBD6VTWVDXVGSSTGL2CLHFWGKZOOEO2MF5OQ, seg_pos={0 0}, upload=648698]: Done uploading piece. err: <nil> Jan 5 12:52:57.283761033 (741h23m57.154844078s) [noise=true , address=193.149.164.143:4703 , piece_id=TYA6X2BSYBRYHJ2WLUENFCLU2ZFC6A2UPWJW3JEEQOAEVFWLTWLQ, seg_pos={0 0}, upload=648698]: Done uploading piece. err: <nil> Jan 5 12:52:57.435217930 (741h23m57.306300965s) [noise=true , address=145.79.227.43:4177 , piece_id=PXINTEEBYFDYXSF2BTQ5ZLJDZ7IKIJOBBQPQUKQD26QXUT4RZ3UA, seg_pos={0 0}, upload=648698]: Done uploading piece. err: <nil> Jan 5 12:52:57.552865089 (741h23m57.423948124s) [noise=true , address=145.79.227.40:4053 , piece_id=RVRFEMIOPASHB5NRRUWDE6T36L3NCR7T7GECNA3BLF76X3RLVI6A, seg_pos={0 0}, upload=648698]: Done uploading piece. err: <nil> Jan 5 12:52:57.602358332 (741h23m57.473441377s) [noise=true , address=77.90.31.10:4542 , piece_id=LEU44EKI74PF6L62ELQVNKH6S3RLI2QWOZIZBMMCML76NTLWSJDQ, seg_pos={0 0}, upload=648698]: Done uploading piece. err: <nil> Jan 5 12:52:57.807263873 (741h23m57.678346918s) [noise=true , address=64.49.13.155:4923 , piece_id=UBNCIHTXJJFCQQGRHG2GFFIYVYBN2K544HSIBFMPBTQUQ3BLCDRQ, seg_pos={0 0}, upload=648698]: Done uploading piece. err: <nil> Jan 5 12:52:58.075551093 (741h23m57.946634128s) [noise=true , address=77.90.31.15:4716 , piece_id=FTUXZUD2VOFOBY42VU77O2KZ2Q5J6ETABI754OX4E5EVQUDH4ILQ, seg_pos={0 0}, upload=648698]: Done uploading piece. err: <nil> Jan 5 12:52:58.075570641 (741h23m57.946653676s) [seg_pos={0 0}, upload=648698]: Segment reached optimal threshold of 14 pieces. Jan 5 12:52:58.075726818 (741h23m57.946809853s) [noise=true , address=45.91.116.14:4201 , piece_id=B4RBSL2C3K7LDTCJKMVIVABOFUVT3NQNDTJXDV4X6NRTQDUSTFTQ, seg_pos={0 0}, upload=648698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RUqtJWMppPz7u7bTYm5chvDf53HVDPCwC7bnmTs1R5hJD8Chp): protocol: expected piece hash; context canceled; EOF Jan 5 12:52:58.075732609 (741h23m57.946815644s) [noise=true , address=45.91.116.14:4201 , piece_id=B4RBSL2C3K7LDTCJKMVIVABOFUVT3NQNDTJXDV4X6NRTQDUSTFTQ, seg_pos={0 0}, upload=648698]: Long tail cancellation due to optimal threshold Jan 5 12:52:58.075921689 (741h23m57.947004754s) [noise=true , address=64.49.13.169:4711 , piece_id=JRWRCUZZW25F7HEAGGPZQU3CG6QOF33S34ZR54WTD4L2N6S675MA, seg_pos={0 0}, upload=648698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52386->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52386->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52386->64.49.13.169:4711: use of closed network connection Jan 5 12:52:58.075933051 (741h23m57.947016096s) [noise=true , address=64.49.13.169:4711 , piece_id=JRWRCUZZW25F7HEAGGPZQU3CG6QOF33S34ZR54WTD4L2N6S675MA, seg_pos={0 0}, upload=648698]: Long tail cancellation due to optimal threshold Jan 5 12:52:58.075956295 (741h23m57.94703933s) [noise=true , address=45.91.116.8:4064 , piece_id=FNB4SN5DOBWXG23BKHN5EOACLXI3NYGOM75Q5YTWGQDJYYVWPO4Q, seg_pos={0 0}, upload=648698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn): protocol: expected piece hash; context canceled; EOF Jan 5 12:52:58.075962797 (741h23m57.947045832s) [noise=true , address=45.91.116.8:4064 , piece_id=FNB4SN5DOBWXG23BKHN5EOACLXI3NYGOM75Q5YTWGQDJYYVWPO4Q, seg_pos={0 0}, upload=648698]: Long tail cancellation due to optimal threshold Jan 5 12:52:58.076097123 (741h23m57.947180158s) [noise=true , address=45.91.116.12:4230 , piece_id=TKXW6FBC4AB4CHGUBK4O67PCQKAERO637ZCRWRLNWS4J3B5HOJKA, seg_pos={0 0}, upload=648698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RqMv3h2jS5V4nVqPmi3BDwT9gdJBC93EvyA8VXf5w77yiAQP): protocol: read tcp 188.166.225.170:45146->45.91.116.12:4230: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 5 12:52:58.076102684 (741h23m57.947185729s) [noise=true , address=45.91.116.12:4230 , piece_id=TKXW6FBC4AB4CHGUBK4O67PCQKAERO637ZCRWRLNWS4J3B5HOJKA, seg_pos={0 0}, upload=648698]: Long tail cancellation due to optimal threshold Jan 5 12:52:58.076158400 (741h23m57.947241435s) [seg_pos={0 0}, upload=648698]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 5 12:52:58.076217442 (741h23m57.947300487s) [upload=648698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:52:58.076221399 (741h23m57.947304444s) [upload=648698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:52:58.366427206 (741h23m58.237510241s) [upload=648698]: Done uploading segments... Jan 5 12:52:58.366448596 (741h23m58.237531631s) [upload=648698]: Upload finished. err: <nil> Duration: 31.090445107s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX....X] duration: 5.001334393s trace: https://jaeger.dev.storj.io/trace/91fdaef13b18fd FAIL (2026-01-05 12:53:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001364129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........................X............................X...X] duration: 1m0.001061592s trace: https://jaeger.dev.storj.io/trace/42bcf85cc84901ec FAIL (2026-01-05 12:52:55 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001085268s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XX.XX.X....X....X..XX..X.XXX.X.XXX....X....XX.....XX.XXX...] duration: 15.431850925s trace: https://jaeger.dev.storj.io/trace/4a6a7727c4f6c3f5 FAIL (2026-01-05 12:52:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== s) [noise=true , address=77.90.31.13:4646 , piece_id=ULJL6M624BP3GU6TCIOC5V3ZWUETQIWLA2UW2CHNOEM2FCRJSUQA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.343860961 (741h21m16.214944006s) [noise=true , address=145.79.227.40:4082 , piece_id=KHAPKFLEILTABN5UKV72AZTA2JVZPMPZH6MUXLN7ZELCJJGZ6B6A, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.344273286 (741h21m16.215356331s) [noise=true , address=45.91.116.7:4030 , piece_id=VEQ6MHDMIWQ7XVFAAQCMXHKZOYPG236ATKPXUL2ZVVZW2CPK7UAA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.344661705 (741h21m16.21574475s) [noise=true , address=64.49.13.169:4711 , piece_id=2AAUF5CA5TTAOLF3HKTAPKZC5OBRVUH4PGUORKTRY3TNYDIT33CQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.345057329 (741h21m16.216140374s) [noise=true , address=45.91.116.8:4058 , piece_id=O5MFGJNQ72FAYKWKCEOH5LFGKAEAHAPGZVRP5XC33ARMGBPSOCDQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.345449235 (741h21m16.21653228s) [noise=true , address=145.79.227.30:4778 , piece_id=M7H36O3C3QS4EMBOJWTCV7NFBQ3CJD7KPYSZDPDBJSOTNAISOHTA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.345943126 (741h21m16.217026171s) [noise=true , address=145.79.227.32:4865 , piece_id=JDM6PXE7QU24DJOKTO4FZZXDTU5TQPPN2WTKTT5OJBVGBRY53YVQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.346612109 (741h21m16.217695154s) [noise=true , address=145.79.227.33:4887 , piece_id=UKEVJ6TE3XMSAZRGCX4SN7QC6RC65WAG7PJE5KKPOOVCWBWTCLSA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.347310950 (741h21m16.218393985s) [noise=true , address=77.90.31.14:4692 , piece_id=M2S5GNMVNGYTCAQAAEUHXGKAYG67OJSSEABX5KMMFHBPDR5Z3VFQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.347901194 (741h21m16.218984239s) [noise=true , address=77.90.31.17:4787 , piece_id=W6A25LZ4IYALHUBQN5LHX243LH3QU5WJ6PBRCDKLJPSKHS6EXXUA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.348497509 (741h21m16.219580554s) [noise=true , address=77.90.31.21:4901 , piece_id=HMYASRS73BYOWRSZFZCKMXPZMPN34FFGO6WEPJ3XIGRZOLK4D7RA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.349002050 (741h21m16.220085086s) [noise=true , address=193.149.164.143:4697 , piece_id=FCY3CJEGPC3HB6VYWWRM3NH2ZOYUOIYR4U3NDKEAANU67N2PBRPQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.349560444 (741h21m16.220643489s) [noise=true , address=145.79.227.18:4008 , piece_id=WL5DFHWAQWI4JC64EZAHCF4GHMTQO6FPPB747ZIZII3OCK6DX6BA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.350173061 (741h21m16.221256096s) [noise=true , address=145.79.227.35:4487 , piece_id=I2LPQMRJUIX63BYV3UQKOJA2G7O6CZLSCMLHL5EQOM7AHPQTSVUA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.351191480 (741h21m16.222274515s) [noise=true , address=145.79.227.29:4962 , piece_id=ANIHIL2ISSBKFS5OBA65NXA2ATOJENYDKYGD32EFRRD6K3GPOYMA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.351708675 (741h21m16.22279171s) [noise=true , address=145.79.227.34:4920 , piece_id=VQ2GR36RVFJJS3DZTR6K6JGWSPOL4BFAUX6LMDBO45YWQ3FGAAGQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:17.767879931 (741h21m17.638962976s) [upload=648658]: Next returned nil segment Jan 5 12:50:17.767896904 (741h21m17.638979949s) [upload=648658]: Waiting for error group managing segments... Jan 5 12:50:18.201707096 (741h21m18.072790141s) [noise=true , address=145.79.227.20:4445 , piece_id=GQ7OZ5GFCRADHOWFQYPOJ4EPQDCAVL4LA5DOXTXEN4CKGALU5IJQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.215283227 (741h21m18.086366262s) [noise=true , address=145.79.227.32:4865 , piece_id=JDM6PXE7QU24DJOKTO4FZZXDTU5TQPPN2WTKTT5OJBVGBRY53YVQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.222243640 (741h21m18.093326685s) [noise=true , address=145.79.227.33:4887 , piece_id=UKEVJ6TE3XMSAZRGCX4SN7QC6RC65WAG7PJE5KKPOOVCWBWTCLSA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.252982321 (741h21m18.124065367s) [noise=true , address=145.79.227.18:4008 , piece_id=WL5DFHWAQWI4JC64EZAHCF4GHMTQO6FPPB747ZIZII3OCK6DX6BA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.850955065 (741h21m18.72203811s) [noise=true , address=77.90.31.21:4901 , piece_id=HMYASRS73BYOWRSZFZCKMXPZMPN34FFGO6WEPJ3XIGRZOLK4D7RA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.031791266 (741h21m18.902874311s) [noise=true , address=193.149.164.143:4697 , piece_id=FCY3CJEGPC3HB6VYWWRM3NH2ZOYUOIYR4U3NDKEAANU67N2PBRPQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.111234969 (741h21m18.982318004s) [noise=true , address=145.79.227.29:4962 , piece_id=ANIHIL2ISSBKFS5OBA65NXA2ATOJENYDKYGD32EFRRD6K3GPOYMA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.117098744 (741h21m18.988181789s) [noise=true , address=145.79.227.40:4082 , piece_id=KHAPKFLEILTABN5UKV72AZTA2JVZPMPZH6MUXLN7ZELCJJGZ6B6A, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.125284711 (741h21m18.996367746s) [noise=true , address=145.79.227.44:4228 , piece_id=TNGVGCOEJSSOL4I4LQABPKAZCODZXCHDIGC2VJK6ZE5JDQG2DSNQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.180346247 (741h21m19.051429292s) [noise=true , address=145.79.227.30:4778 , piece_id=M7H36O3C3QS4EMBOJWTCV7NFBQ3CJD7KPYSZDPDBJSOTNAISOHTA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.240217411 (741h21m19.111300467s) [noise=true , address=77.90.31.17:4787 , piece_id=W6A25LZ4IYALHUBQN5LHX243LH3QU5WJ6PBRCDKLJPSKHS6EXXUA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.392191273 (741h21m19.263274318s) [noise=true , address=145.79.227.34:4920 , piece_id=VQ2GR36RVFJJS3DZTR6K6JGWSPOL4BFAUX6LMDBO45YWQ3FGAAGQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.569174296 (741h21m19.440257341s) [noise=true , address=145.79.227.35:4487 , piece_id=I2LPQMRJUIX63BYV3UQKOJA2G7O6CZLSCMLHL5EQOM7AHPQTSVUA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.791772137 (741h21m19.662855182s) [noise=true , address=45.91.116.7:4030 , piece_id=VEQ6MHDMIWQ7XVFAAQCMXHKZOYPG236ATKPXUL2ZVVZW2CPK7UAA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.791799479 (741h21m19.662882524s) [seg_pos={0 0}, upload=648658]: Segment reached optimal threshold of 14 pieces. Jan 5 12:50:19.792028084 (741h21m19.663111119s) [noise=true , address=64.49.13.169:4711 , piece_id=2AAUF5CA5TTAOLF3HKTAPKZC5OBRVUH4PGUORKTRY3TNYDIT33CQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:53986->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:53986->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53986->64.49.13.169:4711: use of closed network connection Jan 5 12:50:19.792036570 (741h21m19.663119606s) [noise=true , address=64.49.13.169:4711 , piece_id=2AAUF5CA5TTAOLF3HKTAPKZC5OBRVUH4PGUORKTRY3TNYDIT33CQ, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792064985 (741h21m19.66314802s) [noise=true , address=45.91.116.8:4058 , piece_id=O5MFGJNQ72FAYKWKCEOH5LFGKAEAHAPGZVRP5XC33ARMGBPSOCDQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GHuEEAGNMcY8EM4n2kFnvtA7p46ZzNHSZYw7yaXdTRCU1Kq6B): protocol: expected piece hash; context canceled; EOF Jan 5 12:50:19.792075254 (741h21m19.663158289s) [noise=true , address=45.91.116.8:4058 , piece_id=O5MFGJNQ72FAYKWKCEOH5LFGKAEAHAPGZVRP5XC33ARMGBPSOCDQ, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792352893 (741h21m19.663435948s) [noise=true , address=77.90.31.14:4692 , piece_id=M2S5GNMVNGYTCAQAAEUHXGKAYG67OJSSEABX5KMMFHBPDR5Z3VFQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: expected piece hash; context canceled; EOF Jan 5 12:50:19.792365667 (741h21m19.663448702s) [noise=true , address=77.90.31.14:4692 , piece_id=M2S5GNMVNGYTCAQAAEUHXGKAYG67OJSSEABX5KMMFHBPDR5Z3VFQ, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792610503 (741h21m19.663693548s) [noise=true , address=77.90.31.13:4646 , piece_id=ULJL6M624BP3GU6TCIOC5V3ZWUETQIWLA2UW2CHNOEM2FCRJSUQA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: expected piece hash; context canceled; EOF Jan 5 12:50:19.792619219 (741h21m19.663702264s) [noise=true , address=77.90.31.13:4646 , piece_id=ULJL6M624BP3GU6TCIOC5V3ZWUETQIWLA2UW2CHNOEM2FCRJSUQA, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792642774 (741h21m19.663725819s) [seg_pos={0 0}, upload=648658]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 5 12:50:19.792737405 (741h21m19.66382044s) [upload=648658]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:50:19.792743967 (741h21m19.663827012s) [upload=648658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:50:20.082215095 (741h21m19.95329814s) [upload=648658]: Done uploading segments... Jan 5 12:50:20.082251614 (741h21m19.953334659s) [upload=648658]: Upload finished. err: <nil> Duration: 15.431903344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X..X.X..........X...XX...............XX.........X.......X.] duration: 6.728013643s trace: https://jaeger.dev.storj.io/trace/12ee019f05a328f2 FAIL (2026-01-05 12:52:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Jan 5 12:51:48.920874362 (741h22m48.791957397s) [upload=648681]: Starting upload Jan 5 12:51:48.920926541 (741h22m48.792009576s) [upload=648681]: Done starting upload Jan 5 12:51:48.921081566 (741h22m48.792164601s) [upload=648681]: Uploading segments... Jan 5 12:51:48.921179803 (741h22m48.792262838s) [upload=648681]: Got next segment. Inline: false Jan 5 12:51:48.921251730 (741h22m48.792334765s) [upload=648681]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 5 12:51:48.921257271 (741h22m48.792340306s) [upload=648681]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 5 12:51:49.171538276 (741h22m49.042621311s) [seg_pos={0 0}, upload=648681]: Begin upload segment... Jan 5 12:51:49.171591456 (741h22m49.042674491s) [seg_pos={0 0}, upload=648681]: Done begin upload segment. Jan 5 12:51:49.171618979 (741h22m49.042702014s) [noise=true , address=145.79.227.15:4874 , piece_id=NPYLQO7LW7DENESFYTQ7VBWAI56VQ7IBGEAGE4Z5I763RDVKSPHQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.171656450 (741h22m49.042739525s) [noise=true , address=145.79.227.11:4731 , piece_id=F3F6LPZN2MSJTA6WTP272ESURQEST4AAXG3ZZKUVH24MHSEPALBQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.172314674 (741h22m49.043397729s) [noise=true , address=145.79.227.8:4331 , piece_id=ZASY3VD72KZMFSNIWCNDYBZ5R5YHY4UMSJBBEFCUNVWF5MHETUFA, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.172936428 (741h22m49.044019483s) [noise=true , address=45.91.116.11:4118 , piece_id=Q27QBUTQISVNZHH6ATTYCFQZ3UG5JP4CTMEK7MG3SDB6TX4NPU2Q, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.173498688 (741h22m49.044581733s) [noise=true , address=145.79.227.12:4767 , piece_id=F3ENQYNOM3VXAO5QD4YXRHJSENLMBORKO5NUIPBSB44EOW7XJFLQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.175154812 (741h22m49.046237847s) [noise=true , address=145.79.227.46:4602 , piece_id=YLPLZW7MP6AZNSJEFUWHT2IZ5OKPOOQSXIAAIMCGGFL3GF7QBLPQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.176333216 (741h22m49.047416251s) [noise=true , address=45.91.116.9:4261 , piece_id=R3W5WLTCBMKC5ZFEQQNKWJJ37EMM2PDMQAGPI7OJF7YXQA6Y4XBQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.176347714 (741h22m49.047430749s) [noise=true , address=145.79.227.21:4474 , piece_id=E475NZMGOEV75XEVFUIY6QMX65DBDC3FP4Q2DMNV5TQN35OAHNZQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.177406049 (741h22m49.048489084s) [noise=true , address=145.79.227.47:4265 , piece_id=FPMZ3WYUHHO5OZBDSW2XG3XQ3A3KHDB7ZGH4JD533P2PHSHJRBXA, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.178578812 (741h22m49.049661847s) [noise=true , address=45.91.116.12:4226 , piece_id=YYWLWCQBITKCDIGRBLQGPOIOPXT3FRFKOVJW7J45DOCPQRH4BPSQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.178779464 (741h22m49.049862519s) [noise=true , address=45.91.116.14:4192 , piece_id=D6LF7EK2YKY36JTFA74HDFP466Y5N4GP35A752LOTPDEOIWCOKNA, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.179685409 (741h22m49.050768444s) [noise=true , address=145.79.227.26:4655 , piece_id=PUVUWSTGGTAZSRO3GNAHYUWPZXBVVMJIKX3EDZCQWNFFGKCJCOEQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.181154176 (741h22m49.052237211s) [noise=true , address=77.90.31.14:4663 , piece_id=B7SHSCO4A4CHLEJEFXWWOEJGSFXPJHPZNS2XKNFSXICF44FVLBLA, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.182132308 (741h22m49.053215363s) [noise=true , address=145.79.227.3:4149 , piece_id=WI4BPSMNSK4ZH4L6JBSY6QSN4LFCADTZVK5A7ZJHIO564YSCICGA, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.183377850 (741h22m49.054460885s) [noise=true , address=45.91.116.7:4012 , piece_id=MQJB454WCHQOQHGJKBEAJCZGOUCRLTSRH7PWWIHSB3FP3VFUPCYA, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.186851996 (741h22m49.057935041s) [noise=true , address=45.91.116.13:4154 , piece_id=JLOUFMGLO7PC4QOVC7CAOUZBRCZTB6A6FVEIH6NN2YP24D4VXEYA, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:50.845193344 (741h22m50.716276379s) [upload=648681]: Next returned nil segment Jan 5 12:51:50.845213924 (741h22m50.716296959s) [upload=648681]: Waiting for error group managing segments... Jan 5 12:51:51.175349192 (741h22m51.046432237s) [noise=true , address=145.79.227.12:4767 , piece_id=F3ENQYNOM3VXAO5QD4YXRHJSENLMBORKO5NUIPBSB44EOW7XJFLQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.176539589 (741h22m51.047622624s) [noise=true , address=145.79.227.8:4331 , piece_id=ZASY3VD72KZMFSNIWCNDYBZ5R5YHY4UMSJBBEFCUNVWF5MHETUFA, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.177186510 (741h22m51.048269556s) [noise=true , address=145.79.227.15:4874 , piece_id=NPYLQO7LW7DENESFYTQ7VBWAI56VQ7IBGEAGE4Z5I763RDVKSPHQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.178611294 (741h22m51.049694329s) [noise=true , address=145.79.227.47:4265 , piece_id=FPMZ3WYUHHO5OZBDSW2XG3XQ3A3KHDB7ZGH4JD533P2PHSHJRBXA, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.182191882 (741h22m51.053274917s) [noise=true , address=145.79.227.46:4602 , piece_id=YLPLZW7MP6AZNSJEFUWHT2IZ5OKPOOQSXIAAIMCGGFL3GF7QBLPQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.186605166 (741h22m51.057688201s) [noise=true , address=145.79.227.11:4731 , piece_id=F3F6LPZN2MSJTA6WTP272ESURQEST4AAXG3ZZKUVH24MHSEPALBQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.193230030 (741h22m51.064313065s) [noise=true , address=145.79.227.3:4149 , piece_id=WI4BPSMNSK4ZH4L6JBSY6QSN4LFCADTZVK5A7ZJHIO564YSCICGA, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.195397678 (741h22m51.066480713s) [noise=true , address=145.79.227.21:4474 , piece_id=E475NZMGOEV75XEVFUIY6QMX65DBDC3FP4Q2DMNV5TQN35OAHNZQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.200907759 (741h22m51.071990794s) [noise=true , address=145.79.227.26:4655 , piece_id=PUVUWSTGGTAZSRO3GNAHYUWPZXBVVMJIKX3EDZCQWNFFGKCJCOEQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.735053446 (741h22m51.606136481s) [noise=true , address=45.91.116.12:4226 , piece_id=YYWLWCQBITKCDIGRBLQGPOIOPXT3FRFKOVJW7J45DOCPQRH4BPSQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.756959388 (741h22m51.628042423s) [noise=true , address=45.91.116.11:4118 , piece_id=Q27QBUTQISVNZHH6ATTYCFQZ3UG5JP4CTMEK7MG3SDB6TX4NPU2Q, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.941522685 (741h22m51.81260572s) [noise=true , address=77.90.31.14:4663 , piece_id=B7SHSCO4A4CHLEJEFXWWOEJGSFXPJHPZNS2XKNFSXICF44FVLBLA, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:52.017685630 (741h22m51.888768665s) [noise=true , address=45.91.116.7:4012 , piece_id=MQJB454WCHQOQHGJKBEAJCZGOUCRLTSRH7PWWIHSB3FP3VFUPCYA, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:52.017715306 (741h22m51.888798351s) [seg_pos={0 0}, upload=648681]: Segment reached optimal threshold of 13 pieces. Jan 5 12:51:52.017998235 (741h22m51.88908129s) [noise=true , address=45.91.116.14:4192 , piece_id=D6LF7EK2YKY36JTFA74HDFP466Y5N4GP35A752LOTPDEOIWCOKNA, seg_pos={0 0}, upload=648681]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Jan 5 12:51:52.018009977 (741h22m51.889093022s) [noise=true , address=45.91.116.14:4192 , piece_id=D6LF7EK2YKY36JTFA74HDFP466Y5N4GP35A752LOTPDEOIWCOKNA, seg_pos={0 0}, upload=648681]: Long tail cancellation due to optimal threshold Jan 5 12:51:52.018190301 (741h22m51.889273336s) [noise=true , address=45.91.116.9:4261 , piece_id=R3W5WLTCBMKC5ZFEQQNKWJJ37EMM2PDMQAGPI7OJF7YXQA6Y4XBQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:39852->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 5 12:51:52.018200299 (741h22m51.889283354s) [noise=true , address=45.91.116.9:4261 , piece_id=R3W5WLTCBMKC5ZFEQQNKWJJ37EMM2PDMQAGPI7OJF7YXQA6Y4XBQ, seg_pos={0 0}, upload=648681]: Long tail cancellation due to optimal threshold Jan 5 12:51:52.018243402 (741h22m51.889326457s) [noise=true , address=45.91.116.13:4154 , piece_id=JLOUFMGLO7PC4QOVC7CAOUZBRCZTB6A6FVEIH6NN2YP24D4VXEYA, seg_pos={0 0}, upload=648681]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Jan 5 12:51:52.018253872 (741h22m51.889336917s) [noise=true , address=45.91.116.13:4154 , piece_id=JLOUFMGLO7PC4QOVC7CAOUZBRCZTB6A6FVEIH6NN2YP24D4VXEYA, seg_pos={0 0}, upload=648681]: Long tail cancellation due to optimal threshold Jan 5 12:51:52.018279942 (741h22m51.889362977s) [seg_pos={0 0}, upload=648681]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 5 12:51:52.018383890 (741h22m51.889466935s) [upload=648681]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:51:52.018392095 (741h22m51.88947513s) [upload=648681]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:51:52.316532893 (741h22m52.187615938s) [upload=648681]: Done uploading segments... Jan 5 12:51:52.316557329 (741h22m52.187640374s) [upload=648681]: Upload finished. err: <nil> Duration: 6.728046155s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [............X....X.X.....X.....X..X..X...X....XX.X...XXX....] duration: 21.189365903s trace: https://jaeger.dev.storj.io/trace/16381dc260dc9d1a FAIL (2026-01-05 12:52:32 UTC) orbiter warning: context deadline exceeded Duration: 21.189391973s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX....] duration: 4.184258382s trace: https://jaeger.dev.storj.io/trace/2e8e8269d2d4542e FAIL (2026-01-05 12:52:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.184283391s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [................X.............X.............................] duration: 5.533365949s trace: https://jaeger.dev.storj.io/trace/5f0936145b5399cd FAIL (2026-01-05 12:52:26 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.533382852s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................X.............X............................] duration: 21.607253791s trace: https://jaeger.dev.storj.io/trace/5f0936145b5399cd FAIL (2026-01-05 12:52:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 21.607287716s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [..XX.XX.X....X....X..XX..X.XXX.X.XXX....X....XX.....XX.XXX..] duration: 15.269345637s trace: https://jaeger.dev.storj.io/trace/4a6a7727c4f6c3f5 FAIL (2026-01-05 12:52:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== s) [noise=true , address=77.90.31.13:4646 , piece_id=ULJL6M624BP3GU6TCIOC5V3ZWUETQIWLA2UW2CHNOEM2FCRJSUQA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.343860961 (741h21m16.214944006s) [noise=true , address=145.79.227.40:4082 , piece_id=KHAPKFLEILTABN5UKV72AZTA2JVZPMPZH6MUXLN7ZELCJJGZ6B6A, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.344273286 (741h21m16.215356331s) [noise=true , address=45.91.116.7:4030 , piece_id=VEQ6MHDMIWQ7XVFAAQCMXHKZOYPG236ATKPXUL2ZVVZW2CPK7UAA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.344661705 (741h21m16.21574475s) [noise=true , address=64.49.13.169:4711 , piece_id=2AAUF5CA5TTAOLF3HKTAPKZC5OBRVUH4PGUORKTRY3TNYDIT33CQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.345057329 (741h21m16.216140374s) [noise=true , address=45.91.116.8:4058 , piece_id=O5MFGJNQ72FAYKWKCEOH5LFGKAEAHAPGZVRP5XC33ARMGBPSOCDQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.345449235 (741h21m16.21653228s) [noise=true , address=145.79.227.30:4778 , piece_id=M7H36O3C3QS4EMBOJWTCV7NFBQ3CJD7KPYSZDPDBJSOTNAISOHTA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.345943126 (741h21m16.217026171s) [noise=true , address=145.79.227.32:4865 , piece_id=JDM6PXE7QU24DJOKTO4FZZXDTU5TQPPN2WTKTT5OJBVGBRY53YVQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.346612109 (741h21m16.217695154s) [noise=true , address=145.79.227.33:4887 , piece_id=UKEVJ6TE3XMSAZRGCX4SN7QC6RC65WAG7PJE5KKPOOVCWBWTCLSA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.347310950 (741h21m16.218393985s) [noise=true , address=77.90.31.14:4692 , piece_id=M2S5GNMVNGYTCAQAAEUHXGKAYG67OJSSEABX5KMMFHBPDR5Z3VFQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.347901194 (741h21m16.218984239s) [noise=true , address=77.90.31.17:4787 , piece_id=W6A25LZ4IYALHUBQN5LHX243LH3QU5WJ6PBRCDKLJPSKHS6EXXUA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.348497509 (741h21m16.219580554s) [noise=true , address=77.90.31.21:4901 , piece_id=HMYASRS73BYOWRSZFZCKMXPZMPN34FFGO6WEPJ3XIGRZOLK4D7RA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.349002050 (741h21m16.220085086s) [noise=true , address=193.149.164.143:4697 , piece_id=FCY3CJEGPC3HB6VYWWRM3NH2ZOYUOIYR4U3NDKEAANU67N2PBRPQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.349560444 (741h21m16.220643489s) [noise=true , address=145.79.227.18:4008 , piece_id=WL5DFHWAQWI4JC64EZAHCF4GHMTQO6FPPB747ZIZII3OCK6DX6BA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.350173061 (741h21m16.221256096s) [noise=true , address=145.79.227.35:4487 , piece_id=I2LPQMRJUIX63BYV3UQKOJA2G7O6CZLSCMLHL5EQOM7AHPQTSVUA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.351191480 (741h21m16.222274515s) [noise=true , address=145.79.227.29:4962 , piece_id=ANIHIL2ISSBKFS5OBA65NXA2ATOJENYDKYGD32EFRRD6K3GPOYMA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.351708675 (741h21m16.22279171s) [noise=true , address=145.79.227.34:4920 , piece_id=VQ2GR36RVFJJS3DZTR6K6JGWSPOL4BFAUX6LMDBO45YWQ3FGAAGQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:17.767879931 (741h21m17.638962976s) [upload=648658]: Next returned nil segment Jan 5 12:50:17.767896904 (741h21m17.638979949s) [upload=648658]: Waiting for error group managing segments... Jan 5 12:50:18.201707096 (741h21m18.072790141s) [noise=true , address=145.79.227.20:4445 , piece_id=GQ7OZ5GFCRADHOWFQYPOJ4EPQDCAVL4LA5DOXTXEN4CKGALU5IJQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.215283227 (741h21m18.086366262s) [noise=true , address=145.79.227.32:4865 , piece_id=JDM6PXE7QU24DJOKTO4FZZXDTU5TQPPN2WTKTT5OJBVGBRY53YVQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.222243640 (741h21m18.093326685s) [noise=true , address=145.79.227.33:4887 , piece_id=UKEVJ6TE3XMSAZRGCX4SN7QC6RC65WAG7PJE5KKPOOVCWBWTCLSA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.252982321 (741h21m18.124065367s) [noise=true , address=145.79.227.18:4008 , piece_id=WL5DFHWAQWI4JC64EZAHCF4GHMTQO6FPPB747ZIZII3OCK6DX6BA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.850955065 (741h21m18.72203811s) [noise=true , address=77.90.31.21:4901 , piece_id=HMYASRS73BYOWRSZFZCKMXPZMPN34FFGO6WEPJ3XIGRZOLK4D7RA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.031791266 (741h21m18.902874311s) [noise=true , address=193.149.164.143:4697 , piece_id=FCY3CJEGPC3HB6VYWWRM3NH2ZOYUOIYR4U3NDKEAANU67N2PBRPQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.111234969 (741h21m18.982318004s) [noise=true , address=145.79.227.29:4962 , piece_id=ANIHIL2ISSBKFS5OBA65NXA2ATOJENYDKYGD32EFRRD6K3GPOYMA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.117098744 (741h21m18.988181789s) [noise=true , address=145.79.227.40:4082 , piece_id=KHAPKFLEILTABN5UKV72AZTA2JVZPMPZH6MUXLN7ZELCJJGZ6B6A, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.125284711 (741h21m18.996367746s) [noise=true , address=145.79.227.44:4228 , piece_id=TNGVGCOEJSSOL4I4LQABPKAZCODZXCHDIGC2VJK6ZE5JDQG2DSNQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.180346247 (741h21m19.051429292s) [noise=true , address=145.79.227.30:4778 , piece_id=M7H36O3C3QS4EMBOJWTCV7NFBQ3CJD7KPYSZDPDBJSOTNAISOHTA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.240217411 (741h21m19.111300467s) [noise=true , address=77.90.31.17:4787 , piece_id=W6A25LZ4IYALHUBQN5LHX243LH3QU5WJ6PBRCDKLJPSKHS6EXXUA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.392191273 (741h21m19.263274318s) [noise=true , address=145.79.227.34:4920 , piece_id=VQ2GR36RVFJJS3DZTR6K6JGWSPOL4BFAUX6LMDBO45YWQ3FGAAGQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.569174296 (741h21m19.440257341s) [noise=true , address=145.79.227.35:4487 , piece_id=I2LPQMRJUIX63BYV3UQKOJA2G7O6CZLSCMLHL5EQOM7AHPQTSVUA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.791772137 (741h21m19.662855182s) [noise=true , address=45.91.116.7:4030 , piece_id=VEQ6MHDMIWQ7XVFAAQCMXHKZOYPG236ATKPXUL2ZVVZW2CPK7UAA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.791799479 (741h21m19.662882524s) [seg_pos={0 0}, upload=648658]: Segment reached optimal threshold of 14 pieces. Jan 5 12:50:19.792028084 (741h21m19.663111119s) [noise=true , address=64.49.13.169:4711 , piece_id=2AAUF5CA5TTAOLF3HKTAPKZC5OBRVUH4PGUORKTRY3TNYDIT33CQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:53986->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:53986->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53986->64.49.13.169:4711: use of closed network connection Jan 5 12:50:19.792036570 (741h21m19.663119606s) [noise=true , address=64.49.13.169:4711 , piece_id=2AAUF5CA5TTAOLF3HKTAPKZC5OBRVUH4PGUORKTRY3TNYDIT33CQ, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792064985 (741h21m19.66314802s) [noise=true , address=45.91.116.8:4058 , piece_id=O5MFGJNQ72FAYKWKCEOH5LFGKAEAHAPGZVRP5XC33ARMGBPSOCDQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GHuEEAGNMcY8EM4n2kFnvtA7p46ZzNHSZYw7yaXdTRCU1Kq6B): protocol: expected piece hash; context canceled; EOF Jan 5 12:50:19.792075254 (741h21m19.663158289s) [noise=true , address=45.91.116.8:4058 , piece_id=O5MFGJNQ72FAYKWKCEOH5LFGKAEAHAPGZVRP5XC33ARMGBPSOCDQ, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792352893 (741h21m19.663435948s) [noise=true , address=77.90.31.14:4692 , piece_id=M2S5GNMVNGYTCAQAAEUHXGKAYG67OJSSEABX5KMMFHBPDR5Z3VFQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: expected piece hash; context canceled; EOF Jan 5 12:50:19.792365667 (741h21m19.663448702s) [noise=true , address=77.90.31.14:4692 , piece_id=M2S5GNMVNGYTCAQAAEUHXGKAYG67OJSSEABX5KMMFHBPDR5Z3VFQ, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792610503 (741h21m19.663693548s) [noise=true , address=77.90.31.13:4646 , piece_id=ULJL6M624BP3GU6TCIOC5V3ZWUETQIWLA2UW2CHNOEM2FCRJSUQA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: expected piece hash; context canceled; EOF Jan 5 12:50:19.792619219 (741h21m19.663702264s) [noise=true , address=77.90.31.13:4646 , piece_id=ULJL6M624BP3GU6TCIOC5V3ZWUETQIWLA2UW2CHNOEM2FCRJSUQA, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792642774 (741h21m19.663725819s) [seg_pos={0 0}, upload=648658]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 5 12:50:19.792737405 (741h21m19.66382044s) [upload=648658]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:50:19.792743967 (741h21m19.663827012s) [upload=648658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:50:20.082215095 (741h21m19.95329814s) [upload=648658]: Done uploading segments... Jan 5 12:50:20.082251614 (741h21m19.953334659s) [upload=648658]: Upload finished. err: <nil> Duration: 15.269376426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X..X.X..........X...XX...............XX.........X.......X] duration: 30.00334587s trace: https://jaeger.dev.storj.io/trace/12ee019f05a328f2 FAIL (2026-01-05 12:52:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Jan 5 12:51:48.920874362 (741h22m48.791957397s) [upload=648681]: Starting upload Jan 5 12:51:48.920926541 (741h22m48.792009576s) [upload=648681]: Done starting upload Jan 5 12:51:48.921081566 (741h22m48.792164601s) [upload=648681]: Uploading segments... Jan 5 12:51:48.921179803 (741h22m48.792262838s) [upload=648681]: Got next segment. Inline: false Jan 5 12:51:48.921251730 (741h22m48.792334765s) [upload=648681]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 5 12:51:48.921257271 (741h22m48.792340306s) [upload=648681]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 5 12:51:49.171538276 (741h22m49.042621311s) [seg_pos={0 0}, upload=648681]: Begin upload segment... Jan 5 12:51:49.171591456 (741h22m49.042674491s) [seg_pos={0 0}, upload=648681]: Done begin upload segment. Jan 5 12:51:49.171618979 (741h22m49.042702014s) [noise=true , address=145.79.227.15:4874 , piece_id=NPYLQO7LW7DENESFYTQ7VBWAI56VQ7IBGEAGE4Z5I763RDVKSPHQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.171656450 (741h22m49.042739525s) [noise=true , address=145.79.227.11:4731 , piece_id=F3F6LPZN2MSJTA6WTP272ESURQEST4AAXG3ZZKUVH24MHSEPALBQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.172314674 (741h22m49.043397729s) [noise=true , address=145.79.227.8:4331 , piece_id=ZASY3VD72KZMFSNIWCNDYBZ5R5YHY4UMSJBBEFCUNVWF5MHETUFA, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.172936428 (741h22m49.044019483s) [noise=true , address=45.91.116.11:4118 , piece_id=Q27QBUTQISVNZHH6ATTYCFQZ3UG5JP4CTMEK7MG3SDB6TX4NPU2Q, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.173498688 (741h22m49.044581733s) [noise=true , address=145.79.227.12:4767 , piece_id=F3ENQYNOM3VXAO5QD4YXRHJSENLMBORKO5NUIPBSB44EOW7XJFLQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.175154812 (741h22m49.046237847s) [noise=true , address=145.79.227.46:4602 , piece_id=YLPLZW7MP6AZNSJEFUWHT2IZ5OKPOOQSXIAAIMCGGFL3GF7QBLPQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.176333216 (741h22m49.047416251s) [noise=true , address=45.91.116.9:4261 , piece_id=R3W5WLTCBMKC5ZFEQQNKWJJ37EMM2PDMQAGPI7OJF7YXQA6Y4XBQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.176347714 (741h22m49.047430749s) [noise=true , address=145.79.227.21:4474 , piece_id=E475NZMGOEV75XEVFUIY6QMX65DBDC3FP4Q2DMNV5TQN35OAHNZQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.177406049 (741h22m49.048489084s) [noise=true , address=145.79.227.47:4265 , piece_id=FPMZ3WYUHHO5OZBDSW2XG3XQ3A3KHDB7ZGH4JD533P2PHSHJRBXA, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.178578812 (741h22m49.049661847s) [noise=true , address=45.91.116.12:4226 , piece_id=YYWLWCQBITKCDIGRBLQGPOIOPXT3FRFKOVJW7J45DOCPQRH4BPSQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.178779464 (741h22m49.049862519s) [noise=true , address=45.91.116.14:4192 , piece_id=D6LF7EK2YKY36JTFA74HDFP466Y5N4GP35A752LOTPDEOIWCOKNA, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.179685409 (741h22m49.050768444s) [noise=true , address=145.79.227.26:4655 , piece_id=PUVUWSTGGTAZSRO3GNAHYUWPZXBVVMJIKX3EDZCQWNFFGKCJCOEQ, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.181154176 (741h22m49.052237211s) [noise=true , address=77.90.31.14:4663 , piece_id=B7SHSCO4A4CHLEJEFXWWOEJGSFXPJHPZNS2XKNFSXICF44FVLBLA, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.182132308 (741h22m49.053215363s) [noise=true , address=145.79.227.3:4149 , piece_id=WI4BPSMNSK4ZH4L6JBSY6QSN4LFCADTZVK5A7ZJHIO564YSCICGA, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.183377850 (741h22m49.054460885s) [noise=true , address=45.91.116.7:4012 , piece_id=MQJB454WCHQOQHGJKBEAJCZGOUCRLTSRH7PWWIHSB3FP3VFUPCYA, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:49.186851996 (741h22m49.057935041s) [noise=true , address=45.91.116.13:4154 , piece_id=JLOUFMGLO7PC4QOVC7CAOUZBRCZTB6A6FVEIH6NN2YP24D4VXEYA, seg_pos={0 0}, upload=648681]: Uploading piece... Jan 5 12:51:50.845193344 (741h22m50.716276379s) [upload=648681]: Next returned nil segment Jan 5 12:51:50.845213924 (741h22m50.716296959s) [upload=648681]: Waiting for error group managing segments... Jan 5 12:51:51.175349192 (741h22m51.046432237s) [noise=true , address=145.79.227.12:4767 , piece_id=F3ENQYNOM3VXAO5QD4YXRHJSENLMBORKO5NUIPBSB44EOW7XJFLQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.176539589 (741h22m51.047622624s) [noise=true , address=145.79.227.8:4331 , piece_id=ZASY3VD72KZMFSNIWCNDYBZ5R5YHY4UMSJBBEFCUNVWF5MHETUFA, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.177186510 (741h22m51.048269556s) [noise=true , address=145.79.227.15:4874 , piece_id=NPYLQO7LW7DENESFYTQ7VBWAI56VQ7IBGEAGE4Z5I763RDVKSPHQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.178611294 (741h22m51.049694329s) [noise=true , address=145.79.227.47:4265 , piece_id=FPMZ3WYUHHO5OZBDSW2XG3XQ3A3KHDB7ZGH4JD533P2PHSHJRBXA, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.182191882 (741h22m51.053274917s) [noise=true , address=145.79.227.46:4602 , piece_id=YLPLZW7MP6AZNSJEFUWHT2IZ5OKPOOQSXIAAIMCGGFL3GF7QBLPQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.186605166 (741h22m51.057688201s) [noise=true , address=145.79.227.11:4731 , piece_id=F3F6LPZN2MSJTA6WTP272ESURQEST4AAXG3ZZKUVH24MHSEPALBQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.193230030 (741h22m51.064313065s) [noise=true , address=145.79.227.3:4149 , piece_id=WI4BPSMNSK4ZH4L6JBSY6QSN4LFCADTZVK5A7ZJHIO564YSCICGA, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.195397678 (741h22m51.066480713s) [noise=true , address=145.79.227.21:4474 , piece_id=E475NZMGOEV75XEVFUIY6QMX65DBDC3FP4Q2DMNV5TQN35OAHNZQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.200907759 (741h22m51.071990794s) [noise=true , address=145.79.227.26:4655 , piece_id=PUVUWSTGGTAZSRO3GNAHYUWPZXBVVMJIKX3EDZCQWNFFGKCJCOEQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.735053446 (741h22m51.606136481s) [noise=true , address=45.91.116.12:4226 , piece_id=YYWLWCQBITKCDIGRBLQGPOIOPXT3FRFKOVJW7J45DOCPQRH4BPSQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.756959388 (741h22m51.628042423s) [noise=true , address=45.91.116.11:4118 , piece_id=Q27QBUTQISVNZHH6ATTYCFQZ3UG5JP4CTMEK7MG3SDB6TX4NPU2Q, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:51.941522685 (741h22m51.81260572s) [noise=true , address=77.90.31.14:4663 , piece_id=B7SHSCO4A4CHLEJEFXWWOEJGSFXPJHPZNS2XKNFSXICF44FVLBLA, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:52.017685630 (741h22m51.888768665s) [noise=true , address=45.91.116.7:4012 , piece_id=MQJB454WCHQOQHGJKBEAJCZGOUCRLTSRH7PWWIHSB3FP3VFUPCYA, seg_pos={0 0}, upload=648681]: Done uploading piece. err: <nil> Jan 5 12:51:52.017715306 (741h22m51.888798351s) [seg_pos={0 0}, upload=648681]: Segment reached optimal threshold of 13 pieces. Jan 5 12:51:52.017998235 (741h22m51.88908129s) [noise=true , address=45.91.116.14:4192 , piece_id=D6LF7EK2YKY36JTFA74HDFP466Y5N4GP35A752LOTPDEOIWCOKNA, seg_pos={0 0}, upload=648681]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Jan 5 12:51:52.018009977 (741h22m51.889093022s) [noise=true , address=45.91.116.14:4192 , piece_id=D6LF7EK2YKY36JTFA74HDFP466Y5N4GP35A752LOTPDEOIWCOKNA, seg_pos={0 0}, upload=648681]: Long tail cancellation due to optimal threshold Jan 5 12:51:52.018190301 (741h22m51.889273336s) [noise=true , address=45.91.116.9:4261 , piece_id=R3W5WLTCBMKC5ZFEQQNKWJJ37EMM2PDMQAGPI7OJF7YXQA6Y4XBQ, seg_pos={0 0}, upload=648681]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:39852->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 5 12:51:52.018200299 (741h22m51.889283354s) [noise=true , address=45.91.116.9:4261 , piece_id=R3W5WLTCBMKC5ZFEQQNKWJJ37EMM2PDMQAGPI7OJF7YXQA6Y4XBQ, seg_pos={0 0}, upload=648681]: Long tail cancellation due to optimal threshold Jan 5 12:51:52.018243402 (741h22m51.889326457s) [noise=true , address=45.91.116.13:4154 , piece_id=JLOUFMGLO7PC4QOVC7CAOUZBRCZTB6A6FVEIH6NN2YP24D4VXEYA, seg_pos={0 0}, upload=648681]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Jan 5 12:51:52.018253872 (741h22m51.889336917s) [noise=true , address=45.91.116.13:4154 , piece_id=JLOUFMGLO7PC4QOVC7CAOUZBRCZTB6A6FVEIH6NN2YP24D4VXEYA, seg_pos={0 0}, upload=648681]: Long tail cancellation due to optimal threshold Jan 5 12:51:52.018279942 (741h22m51.889362977s) [seg_pos={0 0}, upload=648681]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 5 12:51:52.018383890 (741h22m51.889466935s) [upload=648681]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:51:52.018392095 (741h22m51.88947513s) [upload=648681]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:51:52.316532893 (741h22m52.187615938s) [upload=648681]: Done uploading segments... Jan 5 12:51:52.316557329 (741h22m52.187640374s) [upload=648681]: Upload finished. err: <nil> Duration: 30.003371059s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................X.............X...........................] duration: 6.90454556s trace: https://jaeger.dev.storj.io/trace/5f0936145b5399cd FAIL (2026-01-05 12:51:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.904574925s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX...] duration: 3.615701806s trace: https://jaeger.dev.storj.io/trace/2e8e8269d2d4542e FAIL (2026-01-05 12:51:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.615727204s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X............................X...] duration: 12.641342349s trace: https://jaeger.dev.storj.io/trace/53ac6558e03cb9af FAIL (2026-01-05 12:51:31 UTC) orbiter warning: context deadline exceeded Duration: 12.641368608s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.............X....X.X.....X.....X..X..X...X....XX.X...XXX...] duration: 21.842296194s trace: https://jaeger.dev.storj.io/trace/16381dc260dc9d1a FAIL (2026-01-05 12:51:24 UTC) orbiter warning: context deadline exceeded Duration: 21.842322523s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [...XX.XX.X....X....X..XX..X.XXX.X.XXX....X....XX.....XX.XXX.] duration: 15.460365973s trace: https://jaeger.dev.storj.io/trace/4a6a7727c4f6c3f5 FAIL (2026-01-05 12:51:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== s) [noise=true , address=77.90.31.13:4646 , piece_id=ULJL6M624BP3GU6TCIOC5V3ZWUETQIWLA2UW2CHNOEM2FCRJSUQA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.343860961 (741h21m16.214944006s) [noise=true , address=145.79.227.40:4082 , piece_id=KHAPKFLEILTABN5UKV72AZTA2JVZPMPZH6MUXLN7ZELCJJGZ6B6A, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.344273286 (741h21m16.215356331s) [noise=true , address=45.91.116.7:4030 , piece_id=VEQ6MHDMIWQ7XVFAAQCMXHKZOYPG236ATKPXUL2ZVVZW2CPK7UAA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.344661705 (741h21m16.21574475s) [noise=true , address=64.49.13.169:4711 , piece_id=2AAUF5CA5TTAOLF3HKTAPKZC5OBRVUH4PGUORKTRY3TNYDIT33CQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.345057329 (741h21m16.216140374s) [noise=true , address=45.91.116.8:4058 , piece_id=O5MFGJNQ72FAYKWKCEOH5LFGKAEAHAPGZVRP5XC33ARMGBPSOCDQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.345449235 (741h21m16.21653228s) [noise=true , address=145.79.227.30:4778 , piece_id=M7H36O3C3QS4EMBOJWTCV7NFBQ3CJD7KPYSZDPDBJSOTNAISOHTA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.345943126 (741h21m16.217026171s) [noise=true , address=145.79.227.32:4865 , piece_id=JDM6PXE7QU24DJOKTO4FZZXDTU5TQPPN2WTKTT5OJBVGBRY53YVQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.346612109 (741h21m16.217695154s) [noise=true , address=145.79.227.33:4887 , piece_id=UKEVJ6TE3XMSAZRGCX4SN7QC6RC65WAG7PJE5KKPOOVCWBWTCLSA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.347310950 (741h21m16.218393985s) [noise=true , address=77.90.31.14:4692 , piece_id=M2S5GNMVNGYTCAQAAEUHXGKAYG67OJSSEABX5KMMFHBPDR5Z3VFQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.347901194 (741h21m16.218984239s) [noise=true , address=77.90.31.17:4787 , piece_id=W6A25LZ4IYALHUBQN5LHX243LH3QU5WJ6PBRCDKLJPSKHS6EXXUA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.348497509 (741h21m16.219580554s) [noise=true , address=77.90.31.21:4901 , piece_id=HMYASRS73BYOWRSZFZCKMXPZMPN34FFGO6WEPJ3XIGRZOLK4D7RA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.349002050 (741h21m16.220085086s) [noise=true , address=193.149.164.143:4697 , piece_id=FCY3CJEGPC3HB6VYWWRM3NH2ZOYUOIYR4U3NDKEAANU67N2PBRPQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.349560444 (741h21m16.220643489s) [noise=true , address=145.79.227.18:4008 , piece_id=WL5DFHWAQWI4JC64EZAHCF4GHMTQO6FPPB747ZIZII3OCK6DX6BA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.350173061 (741h21m16.221256096s) [noise=true , address=145.79.227.35:4487 , piece_id=I2LPQMRJUIX63BYV3UQKOJA2G7O6CZLSCMLHL5EQOM7AHPQTSVUA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.351191480 (741h21m16.222274515s) [noise=true , address=145.79.227.29:4962 , piece_id=ANIHIL2ISSBKFS5OBA65NXA2ATOJENYDKYGD32EFRRD6K3GPOYMA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.351708675 (741h21m16.22279171s) [noise=true , address=145.79.227.34:4920 , piece_id=VQ2GR36RVFJJS3DZTR6K6JGWSPOL4BFAUX6LMDBO45YWQ3FGAAGQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:17.767879931 (741h21m17.638962976s) [upload=648658]: Next returned nil segment Jan 5 12:50:17.767896904 (741h21m17.638979949s) [upload=648658]: Waiting for error group managing segments... Jan 5 12:50:18.201707096 (741h21m18.072790141s) [noise=true , address=145.79.227.20:4445 , piece_id=GQ7OZ5GFCRADHOWFQYPOJ4EPQDCAVL4LA5DOXTXEN4CKGALU5IJQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.215283227 (741h21m18.086366262s) [noise=true , address=145.79.227.32:4865 , piece_id=JDM6PXE7QU24DJOKTO4FZZXDTU5TQPPN2WTKTT5OJBVGBRY53YVQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.222243640 (741h21m18.093326685s) [noise=true , address=145.79.227.33:4887 , piece_id=UKEVJ6TE3XMSAZRGCX4SN7QC6RC65WAG7PJE5KKPOOVCWBWTCLSA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.252982321 (741h21m18.124065367s) [noise=true , address=145.79.227.18:4008 , piece_id=WL5DFHWAQWI4JC64EZAHCF4GHMTQO6FPPB747ZIZII3OCK6DX6BA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.850955065 (741h21m18.72203811s) [noise=true , address=77.90.31.21:4901 , piece_id=HMYASRS73BYOWRSZFZCKMXPZMPN34FFGO6WEPJ3XIGRZOLK4D7RA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.031791266 (741h21m18.902874311s) [noise=true , address=193.149.164.143:4697 , piece_id=FCY3CJEGPC3HB6VYWWRM3NH2ZOYUOIYR4U3NDKEAANU67N2PBRPQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.111234969 (741h21m18.982318004s) [noise=true , address=145.79.227.29:4962 , piece_id=ANIHIL2ISSBKFS5OBA65NXA2ATOJENYDKYGD32EFRRD6K3GPOYMA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.117098744 (741h21m18.988181789s) [noise=true , address=145.79.227.40:4082 , piece_id=KHAPKFLEILTABN5UKV72AZTA2JVZPMPZH6MUXLN7ZELCJJGZ6B6A, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.125284711 (741h21m18.996367746s) [noise=true , address=145.79.227.44:4228 , piece_id=TNGVGCOEJSSOL4I4LQABPKAZCODZXCHDIGC2VJK6ZE5JDQG2DSNQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.180346247 (741h21m19.051429292s) [noise=true , address=145.79.227.30:4778 , piece_id=M7H36O3C3QS4EMBOJWTCV7NFBQ3CJD7KPYSZDPDBJSOTNAISOHTA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.240217411 (741h21m19.111300467s) [noise=true , address=77.90.31.17:4787 , piece_id=W6A25LZ4IYALHUBQN5LHX243LH3QU5WJ6PBRCDKLJPSKHS6EXXUA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.392191273 (741h21m19.263274318s) [noise=true , address=145.79.227.34:4920 , piece_id=VQ2GR36RVFJJS3DZTR6K6JGWSPOL4BFAUX6LMDBO45YWQ3FGAAGQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.569174296 (741h21m19.440257341s) [noise=true , address=145.79.227.35:4487 , piece_id=I2LPQMRJUIX63BYV3UQKOJA2G7O6CZLSCMLHL5EQOM7AHPQTSVUA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.791772137 (741h21m19.662855182s) [noise=true , address=45.91.116.7:4030 , piece_id=VEQ6MHDMIWQ7XVFAAQCMXHKZOYPG236ATKPXUL2ZVVZW2CPK7UAA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.791799479 (741h21m19.662882524s) [seg_pos={0 0}, upload=648658]: Segment reached optimal threshold of 14 pieces. Jan 5 12:50:19.792028084 (741h21m19.663111119s) [noise=true , address=64.49.13.169:4711 , piece_id=2AAUF5CA5TTAOLF3HKTAPKZC5OBRVUH4PGUORKTRY3TNYDIT33CQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:53986->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:53986->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53986->64.49.13.169:4711: use of closed network connection Jan 5 12:50:19.792036570 (741h21m19.663119606s) [noise=true , address=64.49.13.169:4711 , piece_id=2AAUF5CA5TTAOLF3HKTAPKZC5OBRVUH4PGUORKTRY3TNYDIT33CQ, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792064985 (741h21m19.66314802s) [noise=true , address=45.91.116.8:4058 , piece_id=O5MFGJNQ72FAYKWKCEOH5LFGKAEAHAPGZVRP5XC33ARMGBPSOCDQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GHuEEAGNMcY8EM4n2kFnvtA7p46ZzNHSZYw7yaXdTRCU1Kq6B): protocol: expected piece hash; context canceled; EOF Jan 5 12:50:19.792075254 (741h21m19.663158289s) [noise=true , address=45.91.116.8:4058 , piece_id=O5MFGJNQ72FAYKWKCEOH5LFGKAEAHAPGZVRP5XC33ARMGBPSOCDQ, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792352893 (741h21m19.663435948s) [noise=true , address=77.90.31.14:4692 , piece_id=M2S5GNMVNGYTCAQAAEUHXGKAYG67OJSSEABX5KMMFHBPDR5Z3VFQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: expected piece hash; context canceled; EOF Jan 5 12:50:19.792365667 (741h21m19.663448702s) [noise=true , address=77.90.31.14:4692 , piece_id=M2S5GNMVNGYTCAQAAEUHXGKAYG67OJSSEABX5KMMFHBPDR5Z3VFQ, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792610503 (741h21m19.663693548s) [noise=true , address=77.90.31.13:4646 , piece_id=ULJL6M624BP3GU6TCIOC5V3ZWUETQIWLA2UW2CHNOEM2FCRJSUQA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: expected piece hash; context canceled; EOF Jan 5 12:50:19.792619219 (741h21m19.663702264s) [noise=true , address=77.90.31.13:4646 , piece_id=ULJL6M624BP3GU6TCIOC5V3ZWUETQIWLA2UW2CHNOEM2FCRJSUQA, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792642774 (741h21m19.663725819s) [seg_pos={0 0}, upload=648658]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 5 12:50:19.792737405 (741h21m19.66382044s) [upload=648658]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:50:19.792743967 (741h21m19.663827012s) [upload=648658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:50:20.082215095 (741h21m19.95329814s) [upload=648658]: Done uploading segments... Jan 5 12:50:20.082251614 (741h21m19.953334659s) [upload=648658]: Upload finished. err: <nil> Duration: 15.460382074s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................X.............X..........................] duration: 5.841731288s trace: https://jaeger.dev.storj.io/trace/5f0936145b5399cd FAIL (2026-01-05 12:51:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.841757058s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X...X..X.X..........X...XX...............XX.........X.......] duration: 6.376813578s trace: https://jaeger.dev.storj.io/trace/28cbbf043d214b8f FAIL (2026-01-05 12:51:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5 12:45:21.016391497 (741h16m20.887474532s) [upload=648589]: Starting upload Jan 5 12:45:21.016449508 (741h16m20.887532543s) [upload=648589]: Done starting upload Jan 5 12:45:21.016606537 (741h16m20.887689572s) [upload=648589]: Uploading segments... Jan 5 12:45:21.016706537 (741h16m20.887789582s) [upload=648589]: Got next segment. Inline: false Jan 5 12:45:21.016739780 (741h16m20.887822825s) [upload=648589]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 5 12:45:21.016744811 (741h16m20.887827846s) [upload=648589]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 5 12:45:21.268595884 (741h16m21.139678929s) [seg_pos={0 0}, upload=648589]: Begin upload segment... Jan 5 12:45:21.268642443 (741h16m21.139725488s) [seg_pos={0 0}, upload=648589]: Done begin upload segment. Jan 5 12:45:21.268730540 (741h16m21.139813575s) [noise=true , address=145.79.227.22:4509 , piece_id=25JPBK3T7REYJP2WLO5K4LBLVIONOTRD5752G7MZWUHNXQ36562Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.270879172 (741h16m21.141962207s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.271442805 (741h16m21.14252584s) [noise=true , address=145.79.227.20:4437 , piece_id=VN5TKKWDYLGS7DBM5JKB6K2USRYFC7FUB4ZFHV3XJCJXZZ756G3Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.271905155 (741h16m21.14298819s) [noise=true , address=145.79.227.26:4653 , piece_id=DCMR6IJGZQELRMNI7LTMUPTCLSSL3U4JFR5TNZG2P36YSRN565TA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.272351776 (741h16m21.143434821s) [noise=true , address=145.79.227.48:4303 , piece_id=2D5IDHWWJXXZZH5WEL5UUDPWCCQJSYLB3HRWUHJAFHIJ6HQC7VZA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.272842941 (741h16m21.143925986s) [noise=true , address=145.79.227.28:4725 , piece_id=RHBNZJYMSWERVLZNUAEYSJNYCBWBKQZL72DX7AXGKU7V5PUAPM3Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.273284582 (741h16m21.144367627s) [noise=true , address=145.79.227.38:4979 , piece_id=BI5R7DQQFUDGR4AKJCXO6VUBKYKAZI5EAF5BWICVCDGD64NPQJQA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.273686899 (741h16m21.144769934s) [noise=true , address=45.91.116.14:4192 , piece_id=EUQFZAAJX5MN7TCT6U4K4VNL2UND56WESFXBDEL4K3FYBKZBNCNQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.274174016 (741h16m21.145257061s) [noise=true , address=145.79.227.50:4374 , piece_id=NFZSOJMZOLZKF7V52F37WAPSN6V7ZFYTPWY43ICQWIKDPPP4MHUA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.274699497 (741h16m21.145782532s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.275224597 (741h16m21.146307632s) [noise=true , address=145.79.227.5:4222 , piece_id=B2LPQWSTK6ESRTVV5LGUJZVXL277TV3FTCN6FFGXGVUGCRPBVYUA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.275731632 (741h16m21.146814677s) [noise=true , address=145.79.227.42:4121 , piece_id=RZNCW5NERG453IQLF564CPOUEEPLFK4GIZUWCV4WHHVZ5DIW66GQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.277290099 (741h16m21.148373134s) [noise=true , address=77.90.31.19:4842 , piece_id=F7E2LUJMCYFWHOGNSM2VXUTTMCFCP37VZQLCLP5VD2FTN5CQY7NQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.279405628 (741h16m21.150488673s) [noise=true , address=77.90.31.18:4808 , piece_id=VLTWI3TUQ5DZWJS7KYATLVDHP6RK3WPRY4I65KVAHJUPHBMOCY7Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.281446874 (741h16m21.152529919s) [noise=true , address=145.79.227.37:4565 , piece_id=VCJGFGJDQSLNSKMOAKDZZ5QFXVPA3DEWRCWFWRT6EIKRHZJZGR5A, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.283204051 (741h16m21.154287096s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:22.966583542 (741h16m22.837666587s) [upload=648589]: Next returned nil segment Jan 5 12:45:22.966601036 (741h16m22.837684081s) [upload=648589]: Waiting for error group managing segments... Jan 5 12:45:23.259998799 (741h16m23.131081834s) [noise=true , address=145.79.227.22:4509 , piece_id=25JPBK3T7REYJP2WLO5K4LBLVIONOTRD5752G7MZWUHNXQ36562Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.266672608 (741h16m23.137755653s) [noise=true , address=145.79.227.28:4725 , piece_id=RHBNZJYMSWERVLZNUAEYSJNYCBWBKQZL72DX7AXGKU7V5PUAPM3Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.268418462 (741h16m23.139501507s) [noise=true , address=145.79.227.5:4222 , piece_id=B2LPQWSTK6ESRTVV5LGUJZVXL277TV3FTCN6FFGXGVUGCRPBVYUA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.275051953 (741h16m23.146134998s) [noise=true , address=145.79.227.38:4979 , piece_id=BI5R7DQQFUDGR4AKJCXO6VUBKYKAZI5EAF5BWICVCDGD64NPQJQA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.277622617 (741h16m23.148705662s) [noise=true , address=145.79.227.37:4565 , piece_id=VCJGFGJDQSLNSKMOAKDZZ5QFXVPA3DEWRCWFWRT6EIKRHZJZGR5A, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.288675784 (741h16m23.159758819s) [noise=true , address=145.79.227.20:4437 , piece_id=VN5TKKWDYLGS7DBM5JKB6K2USRYFC7FUB4ZFHV3XJCJXZZ756G3Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.288731430 (741h16m23.159814475s) [noise=true , address=145.79.227.48:4303 , piece_id=2D5IDHWWJXXZZH5WEL5UUDPWCCQJSYLB3HRWUHJAFHIJ6HQC7VZA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.290197542 (741h16m23.161280587s) [noise=true , address=145.79.227.26:4653 , piece_id=DCMR6IJGZQELRMNI7LTMUPTCLSSL3U4JFR5TNZG2P36YSRN565TA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.291884975 (741h16m23.16296803s) [noise=true , address=145.79.227.50:4374 , piece_id=NFZSOJMZOLZKF7V52F37WAPSN6V7ZFYTPWY43ICQWIKDPPP4MHUA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.292911229 (741h16m23.163994274s) [noise=true , address=145.79.227.42:4121 , piece_id=RZNCW5NERG453IQLF564CPOUEEPLFK4GIZUWCV4WHHVZ5DIW66GQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.808597866 (741h16m23.679680911s) [noise=true , address=45.91.116.14:4192 , piece_id=EUQFZAAJX5MN7TCT6U4K4VNL2UND56WESFXBDEL4K3FYBKZBNCNQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:24.376048792 (741h16m24.247131837s) [noise=true , address=77.90.31.18:4808 , piece_id=VLTWI3TUQ5DZWJS7KYATLVDHP6RK3WPRY4I65KVAHJUPHBMOCY7Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:25.047325497 (741h16m24.918408542s) [noise=true , address=77.90.31.19:4842 , piece_id=F7E2LUJMCYFWHOGNSM2VXUTTMCFCP37VZQLCLP5VD2FTN5CQY7NQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:25.047352187 (741h16m24.918435222s) [seg_pos={0 0}, upload=648589]: Segment reached optimal threshold of 13 pieces. Jan 5 12:45:25.047587886 (741h16m24.918670931s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jan 5 12:45:25.047597243 (741h16m24.918680288s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047601892 (741h16m24.918684947s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection Jan 5 12:45:25.047610149 (741h16m24.918693194s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047827893 (741h16m24.918910938s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Jan 5 12:45:25.047837501 (741h16m24.918920546s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047856027 (741h16m24.918939072s) [seg_pos={0 0}, upload=648589]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 5 12:45:25.047959614 (741h16m24.919042659s) [upload=648589]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:45:25.047967589 (741h16m24.919050624s) [upload=648589]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:45:25.345497861 (741h16m25.216580906s) [upload=648589]: Done uploading segments... Jan 5 12:45:25.345532617 (741h16m25.216615662s) [upload=648589]: Upload finished. err: <nil> Duration: 6.376846391s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X............................X..] duration: 25.348077788s trace: https://jaeger.dev.storj.io/trace/53ac6558e03cb9af FAIL (2026-01-05 12:51:01 UTC) orbiter warning: context deadline exceeded Duration: 25.348110831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 26 out of 60 is failed (threshold=3) [X...XX.XX.X....X....X..XX..X.XXX.X.XXX....X....XX.....XX.XXX] duration: 42.712582648s trace: https://jaeger.dev.storj.io/trace/4a6a7727c4f6c3f5 FAIL (2026-01-05 12:50:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== s) [noise=true , address=77.90.31.13:4646 , piece_id=ULJL6M624BP3GU6TCIOC5V3ZWUETQIWLA2UW2CHNOEM2FCRJSUQA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.343860961 (741h21m16.214944006s) [noise=true , address=145.79.227.40:4082 , piece_id=KHAPKFLEILTABN5UKV72AZTA2JVZPMPZH6MUXLN7ZELCJJGZ6B6A, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.344273286 (741h21m16.215356331s) [noise=true , address=45.91.116.7:4030 , piece_id=VEQ6MHDMIWQ7XVFAAQCMXHKZOYPG236ATKPXUL2ZVVZW2CPK7UAA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.344661705 (741h21m16.21574475s) [noise=true , address=64.49.13.169:4711 , piece_id=2AAUF5CA5TTAOLF3HKTAPKZC5OBRVUH4PGUORKTRY3TNYDIT33CQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.345057329 (741h21m16.216140374s) [noise=true , address=45.91.116.8:4058 , piece_id=O5MFGJNQ72FAYKWKCEOH5LFGKAEAHAPGZVRP5XC33ARMGBPSOCDQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.345449235 (741h21m16.21653228s) [noise=true , address=145.79.227.30:4778 , piece_id=M7H36O3C3QS4EMBOJWTCV7NFBQ3CJD7KPYSZDPDBJSOTNAISOHTA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.345943126 (741h21m16.217026171s) [noise=true , address=145.79.227.32:4865 , piece_id=JDM6PXE7QU24DJOKTO4FZZXDTU5TQPPN2WTKTT5OJBVGBRY53YVQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.346612109 (741h21m16.217695154s) [noise=true , address=145.79.227.33:4887 , piece_id=UKEVJ6TE3XMSAZRGCX4SN7QC6RC65WAG7PJE5KKPOOVCWBWTCLSA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.347310950 (741h21m16.218393985s) [noise=true , address=77.90.31.14:4692 , piece_id=M2S5GNMVNGYTCAQAAEUHXGKAYG67OJSSEABX5KMMFHBPDR5Z3VFQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.347901194 (741h21m16.218984239s) [noise=true , address=77.90.31.17:4787 , piece_id=W6A25LZ4IYALHUBQN5LHX243LH3QU5WJ6PBRCDKLJPSKHS6EXXUA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.348497509 (741h21m16.219580554s) [noise=true , address=77.90.31.21:4901 , piece_id=HMYASRS73BYOWRSZFZCKMXPZMPN34FFGO6WEPJ3XIGRZOLK4D7RA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.349002050 (741h21m16.220085086s) [noise=true , address=193.149.164.143:4697 , piece_id=FCY3CJEGPC3HB6VYWWRM3NH2ZOYUOIYR4U3NDKEAANU67N2PBRPQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.349560444 (741h21m16.220643489s) [noise=true , address=145.79.227.18:4008 , piece_id=WL5DFHWAQWI4JC64EZAHCF4GHMTQO6FPPB747ZIZII3OCK6DX6BA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.350173061 (741h21m16.221256096s) [noise=true , address=145.79.227.35:4487 , piece_id=I2LPQMRJUIX63BYV3UQKOJA2G7O6CZLSCMLHL5EQOM7AHPQTSVUA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.351191480 (741h21m16.222274515s) [noise=true , address=145.79.227.29:4962 , piece_id=ANIHIL2ISSBKFS5OBA65NXA2ATOJENYDKYGD32EFRRD6K3GPOYMA, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:16.351708675 (741h21m16.22279171s) [noise=true , address=145.79.227.34:4920 , piece_id=VQ2GR36RVFJJS3DZTR6K6JGWSPOL4BFAUX6LMDBO45YWQ3FGAAGQ, seg_pos={0 0}, upload=648658]: Uploading piece... Jan 5 12:50:17.767879931 (741h21m17.638962976s) [upload=648658]: Next returned nil segment Jan 5 12:50:17.767896904 (741h21m17.638979949s) [upload=648658]: Waiting for error group managing segments... Jan 5 12:50:18.201707096 (741h21m18.072790141s) [noise=true , address=145.79.227.20:4445 , piece_id=GQ7OZ5GFCRADHOWFQYPOJ4EPQDCAVL4LA5DOXTXEN4CKGALU5IJQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.215283227 (741h21m18.086366262s) [noise=true , address=145.79.227.32:4865 , piece_id=JDM6PXE7QU24DJOKTO4FZZXDTU5TQPPN2WTKTT5OJBVGBRY53YVQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.222243640 (741h21m18.093326685s) [noise=true , address=145.79.227.33:4887 , piece_id=UKEVJ6TE3XMSAZRGCX4SN7QC6RC65WAG7PJE5KKPOOVCWBWTCLSA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.252982321 (741h21m18.124065367s) [noise=true , address=145.79.227.18:4008 , piece_id=WL5DFHWAQWI4JC64EZAHCF4GHMTQO6FPPB747ZIZII3OCK6DX6BA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:18.850955065 (741h21m18.72203811s) [noise=true , address=77.90.31.21:4901 , piece_id=HMYASRS73BYOWRSZFZCKMXPZMPN34FFGO6WEPJ3XIGRZOLK4D7RA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.031791266 (741h21m18.902874311s) [noise=true , address=193.149.164.143:4697 , piece_id=FCY3CJEGPC3HB6VYWWRM3NH2ZOYUOIYR4U3NDKEAANU67N2PBRPQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.111234969 (741h21m18.982318004s) [noise=true , address=145.79.227.29:4962 , piece_id=ANIHIL2ISSBKFS5OBA65NXA2ATOJENYDKYGD32EFRRD6K3GPOYMA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.117098744 (741h21m18.988181789s) [noise=true , address=145.79.227.40:4082 , piece_id=KHAPKFLEILTABN5UKV72AZTA2JVZPMPZH6MUXLN7ZELCJJGZ6B6A, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.125284711 (741h21m18.996367746s) [noise=true , address=145.79.227.44:4228 , piece_id=TNGVGCOEJSSOL4I4LQABPKAZCODZXCHDIGC2VJK6ZE5JDQG2DSNQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.180346247 (741h21m19.051429292s) [noise=true , address=145.79.227.30:4778 , piece_id=M7H36O3C3QS4EMBOJWTCV7NFBQ3CJD7KPYSZDPDBJSOTNAISOHTA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.240217411 (741h21m19.111300467s) [noise=true , address=77.90.31.17:4787 , piece_id=W6A25LZ4IYALHUBQN5LHX243LH3QU5WJ6PBRCDKLJPSKHS6EXXUA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.392191273 (741h21m19.263274318s) [noise=true , address=145.79.227.34:4920 , piece_id=VQ2GR36RVFJJS3DZTR6K6JGWSPOL4BFAUX6LMDBO45YWQ3FGAAGQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.569174296 (741h21m19.440257341s) [noise=true , address=145.79.227.35:4487 , piece_id=I2LPQMRJUIX63BYV3UQKOJA2G7O6CZLSCMLHL5EQOM7AHPQTSVUA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.791772137 (741h21m19.662855182s) [noise=true , address=45.91.116.7:4030 , piece_id=VEQ6MHDMIWQ7XVFAAQCMXHKZOYPG236ATKPXUL2ZVVZW2CPK7UAA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: <nil> Jan 5 12:50:19.791799479 (741h21m19.662882524s) [seg_pos={0 0}, upload=648658]: Segment reached optimal threshold of 14 pieces. Jan 5 12:50:19.792028084 (741h21m19.663111119s) [noise=true , address=64.49.13.169:4711 , piece_id=2AAUF5CA5TTAOLF3HKTAPKZC5OBRVUH4PGUORKTRY3TNYDIT33CQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:53986->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:53986->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53986->64.49.13.169:4711: use of closed network connection Jan 5 12:50:19.792036570 (741h21m19.663119606s) [noise=true , address=64.49.13.169:4711 , piece_id=2AAUF5CA5TTAOLF3HKTAPKZC5OBRVUH4PGUORKTRY3TNYDIT33CQ, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792064985 (741h21m19.66314802s) [noise=true , address=45.91.116.8:4058 , piece_id=O5MFGJNQ72FAYKWKCEOH5LFGKAEAHAPGZVRP5XC33ARMGBPSOCDQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GHuEEAGNMcY8EM4n2kFnvtA7p46ZzNHSZYw7yaXdTRCU1Kq6B): protocol: expected piece hash; context canceled; EOF Jan 5 12:50:19.792075254 (741h21m19.663158289s) [noise=true , address=45.91.116.8:4058 , piece_id=O5MFGJNQ72FAYKWKCEOH5LFGKAEAHAPGZVRP5XC33ARMGBPSOCDQ, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792352893 (741h21m19.663435948s) [noise=true , address=77.90.31.14:4692 , piece_id=M2S5GNMVNGYTCAQAAEUHXGKAYG67OJSSEABX5KMMFHBPDR5Z3VFQ, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ckv57Jo1CoAnLarXfwwQB2qmUNxjVuNksEpiXchpPVGxLfw3T): protocol: expected piece hash; context canceled; EOF Jan 5 12:50:19.792365667 (741h21m19.663448702s) [noise=true , address=77.90.31.14:4692 , piece_id=M2S5GNMVNGYTCAQAAEUHXGKAYG67OJSSEABX5KMMFHBPDR5Z3VFQ, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792610503 (741h21m19.663693548s) [noise=true , address=77.90.31.13:4646 , piece_id=ULJL6M624BP3GU6TCIOC5V3ZWUETQIWLA2UW2CHNOEM2FCRJSUQA, seg_pos={0 0}, upload=648658]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: expected piece hash; context canceled; EOF Jan 5 12:50:19.792619219 (741h21m19.663702264s) [noise=true , address=77.90.31.13:4646 , piece_id=ULJL6M624BP3GU6TCIOC5V3ZWUETQIWLA2UW2CHNOEM2FCRJSUQA, seg_pos={0 0}, upload=648658]: Long tail cancellation due to optimal threshold Jan 5 12:50:19.792642774 (741h21m19.663725819s) [seg_pos={0 0}, upload=648658]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 5 12:50:19.792737405 (741h21m19.66382044s) [upload=648658]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:50:19.792743967 (741h21m19.663827012s) [upload=648658]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:50:20.082215095 (741h21m19.95329814s) [upload=648658]: Done uploading segments... Jan 5 12:50:20.082251614 (741h21m19.953334659s) [upload=648658]: Upload finished. err: <nil> Duration: 42.712648274s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX..] duration: 3.994558984s trace: https://jaeger.dev.storj.io/trace/2e8e8269d2d4542e FAIL (2026-01-05 12:50:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.99458895s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................X.............X.........................] duration: 8.016134695s trace: https://jaeger.dev.storj.io/trace/5f0936145b5399cd FAIL (2026-01-05 12:50:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 8.016159172s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............X....X.X.....X.....X..X..X...X....XX.X...XXX..] duration: 25.435321672s trace: https://jaeger.dev.storj.io/trace/16381dc260dc9d1a FAIL (2026-01-05 12:50:18 UTC) orbiter warning: context deadline exceeded Duration: 25.43533693s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.............X........................] duration: 6.757806602s trace: https://jaeger.dev.storj.io/trace/5f0936145b5399cd FAIL (2026-01-05 12:50:12 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.757840528s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X...XX.XX.X....X....X..XX..X.XXX.X.XXX....X....XX.....XX.XX] duration: 43.421186626s trace: https://jaeger.dev.storj.io/trace/55025ad364f7772 FAIL (2026-01-05 12:50:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 9:19.496969613 (741h20m19.368052648s) [noise=true , address=145.79.227.34:4937 , piece_id=YF3SPF77UYHEWZCF5OSIHOOT3CLATLAGZUQFV6HHKFUJZGXO7XAA, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.497501406 (741h20m19.368584451s) [noise=true , address=145.79.227.29:4967 , piece_id=GTRHAQSAMK7D7L5KC5SNX6ZESE63YNITX6QLTOCSSRSQ5K6IGE3Q, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.497636012 (741h20m19.368719057s) [noise=true , address=64.49.13.155:4924 , piece_id=AHRCZUF67L674EDGP2BNVNT4TZH76MZKFQ6XSTAPPKXZHKO2Z7CA, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.498043679 (741h20m19.369126724s) [noise=true , address=145.79.227.6:4285 , piece_id=JSDDDKI73UC44WITFP2FBUF4346FENQUOHGOFUPEMSHY577JB3AA, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.498466343 (741h20m19.369549388s) [noise=true , address=193.149.164.143:4698 , piece_id=MHQSJRAGJRP5EY24FPQ2GRB63MKUWMC4PAGLONGZ5AIUFFKEO7KQ, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.498834203 (741h20m19.369917248s) [noise=true , address=145.79.227.44:4206 , piece_id=ORKIOELHPZUY35LIGOV5LMFGDPGLVKVW4NLBUPD52K6O2BYPHE4Q, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.499216231 (741h20m19.370299276s) [noise=true , address=45.91.116.5:4322 , piece_id=J4HW7S2RJ4A7OMNHWAV7335DR32URVR5E4JM5Y6YGGSJ7SK3J7EA, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.499576127 (741h20m19.370659172s) [noise=true , address=45.91.116.14:4198 , piece_id=XS7JQ4KPRQ3IDFVA6LNI4NZJOBCI4IO77KK65FVCHETRJGTTKD4Q, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.499803509 (741h20m19.370886554s) [noise=true , address=77.90.31.14:4678 , piece_id=IX26ZLP2BBTKZXP74QRPZYEU67ICWUSGUU2L7XCMIORF5M3Q73KQ, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.500383614 (741h20m19.371466659s) [noise=true , address=145.79.227.40:4065 , piece_id=OTMEUTOQI55UWT24RH3O42IE6WAMGFNLLDJ7PIF2N4AY7NBYPK5Q, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.501123833 (741h20m19.372206878s) [noise=true , address=145.79.227.35:4485 , piece_id=PL2CJBTQPDRFKNHCTGZAVYYJJZNUI7INE7DS6K5QQCXC3AUUN6RQ, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.501562379 (741h20m19.372645424s) [noise=true , address=77.90.31.15:4704 , piece_id=YZ3R4DWZXSDGWLEHSRN3EMV7FFOG7WN2TCKBUHYPC4M3FUMCRJGQ, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.502148996 (741h20m19.373232031s) [noise=true , address=145.79.227.49:4361 , piece_id=BS2WAGVFHPLJTQC4NUSZFB63R4KJ5O5QOA4U2AXAP4RDH4SYA4FQ, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.502515112 (741h20m19.373598158s) [noise=true , address=145.79.227.5:4229 , piece_id=M2DPSKPHNDWCF35CPR6M4GU6NKRR3733O6QMYYKW3HPL4FPTMQUA, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.503054510 (741h20m19.374137545s) [noise=true , address=145.79.227.14:4844 , piece_id=76E3B4PFOL5XVQ3RPVIOAF5WES6MSQSMLKFJ4HT7PPIF45RXSHLQ, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.503489067 (741h20m19.374572102s) [noise=true , address=145.79.227.10:4429 , piece_id=ZQLW5YW3RBRMJVBCCPOKN2BEY5QIT673NL63AKLSDHY57HFQJXYQ, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:19.503917673 (741h20m19.375000708s) [noise=true , address=45.91.116.7:4035 , piece_id=2FXPEQOCEFXQJMT3XHDLY65N6QIMRJOBS67PK3KK7RG5MLS2QVYA, seg_pos={0 0}, upload=648645]: Uploading piece... Jan 5 12:49:20.914909029 (741h20m20.785992064s) [upload=648645]: Next returned nil segment Jan 5 12:49:20.914952131 (741h20m20.786035186s) [upload=648645]: Waiting for error group managing segments... Jan 5 12:49:21.346800737 (741h20m21.217883772s) [noise=true , address=64.49.13.155:4924 , piece_id=AHRCZUF67L674EDGP2BNVNT4TZH76MZKFQ6XSTAPPKXZHKO2Z7CA, seg_pos={0 0}, upload=648645]: Done uploading piece. err: <nil> Jan 5 12:49:21.348785978 (741h20m21.219869023s) [noise=true , address=145.79.227.14:4844 , piece_id=76E3B4PFOL5XVQ3RPVIOAF5WES6MSQSMLKFJ4HT7PPIF45RXSHLQ, seg_pos={0 0}, upload=648645]: Done uploading piece. err: <nil> Jan 5 12:49:21.349723382 (741h20m21.220806427s) [noise=true , address=145.79.227.10:4429 , piece_id=ZQLW5YW3RBRMJVBCCPOKN2BEY5QIT673NL63AKLSDHY57HFQJXYQ, seg_pos={0 0}, upload=648645]: Done uploading piece. err: <nil> Jan 5 12:49:21.359256555 (741h20m21.2303396s) [noise=true , address=145.79.227.5:4229 , piece_id=M2DPSKPHNDWCF35CPR6M4GU6NKRR3733O6QMYYKW3HPL4FPTMQUA, seg_pos={0 0}, upload=648645]: Done uploading piece. err: <nil> Jan 5 12:49:21.415547722 (741h20m21.286630767s) [noise=true , address=145.79.227.29:4967 , piece_id=GTRHAQSAMK7D7L5KC5SNX6ZESE63YNITX6QLTOCSSRSQ5K6IGE3Q, seg_pos={0 0}, upload=648645]: Done uploading piece. err: <nil> Jan 5 12:49:21.758238659 (741h20m21.629321704s) [noise=true , address=77.90.31.14:4678 , piece_id=IX26ZLP2BBTKZXP74QRPZYEU67ICWUSGUU2L7XCMIORF5M3Q73KQ, seg_pos={0 0}, upload=648645]: Done uploading piece. err: <nil> Jan 5 12:49:21.808426985 (741h20m21.67951002s) [noise=true , address=45.91.116.7:4035 , piece_id=2FXPEQOCEFXQJMT3XHDLY65N6QIMRJOBS67PK3KK7RG5MLS2QVYA, seg_pos={0 0}, upload=648645]: Done uploading piece. err: <nil> Jan 5 12:49:21.964777591 (741h20m21.835860636s) [noise=true , address=45.91.116.5:4322 , piece_id=J4HW7S2RJ4A7OMNHWAV7335DR32URVR5E4JM5Y6YGGSJ7SK3J7EA, seg_pos={0 0}, upload=648645]: Done uploading piece. err: <nil> Jan 5 12:49:22.151762479 (741h20m22.022845514s) [noise=true , address=193.149.164.143:4698 , piece_id=MHQSJRAGJRP5EY24FPQ2GRB63MKUWMC4PAGLONGZ5AIUFFKEO7KQ, seg_pos={0 0}, upload=648645]: Done uploading piece. err: <nil> Jan 5 12:49:22.296062288 (741h20m22.167145333s) [noise=true , address=145.79.227.30:4765 , piece_id=2CVOSAWIGAPL3UO4BHVCBCYNH5DTRD3DTWQIGRTC7DR3TEAPRJ4A, seg_pos={0 0}, upload=648645]: Done uploading piece. err: <nil> Jan 5 12:49:22.298616141 (741h20m22.169699176s) [noise=true , address=145.79.227.49:4361 , piece_id=BS2WAGVFHPLJTQC4NUSZFB63R4KJ5O5QOA4U2AXAP4RDH4SYA4FQ, seg_pos={0 0}, upload=648645]: Done uploading piece. err: <nil> Jan 5 12:49:22.309416216 (741h20m22.180499261s) [noise=true , address=145.79.227.40:4065 , piece_id=OTMEUTOQI55UWT24RH3O42IE6WAMGFNLLDJ7PIF2N4AY7NBYPK5Q, seg_pos={0 0}, upload=648645]: Done uploading piece. err: <nil> Jan 5 12:49:22.361044664 (741h20m22.232127699s) [noise=true , address=145.79.227.44:4206 , piece_id=ORKIOELHPZUY35LIGOV5LMFGDPGLVKVW4NLBUPD52K6O2BYPHE4Q, seg_pos={0 0}, upload=648645]: Done uploading piece. err: <nil> Jan 5 12:49:22.620568969 (741h20m22.491652014s) [noise=true , address=45.91.116.14:4198 , piece_id=XS7JQ4KPRQ3IDFVA6LNI4NZJOBCI4IO77KK65FVCHETRJGTTKD4Q, seg_pos={0 0}, upload=648645]: Done uploading piece. err: <nil> Jan 5 12:49:22.620593415 (741h20m22.49167646s) [seg_pos={0 0}, upload=648645]: Segment reached optimal threshold of 14 pieces. Jan 5 12:49:22.620768999 (741h20m22.491852044s) [noise=true , address=145.79.227.35:4485 , piece_id=PL2CJBTQPDRFKNHCTGZAVYYJJZNUI7INE7DS6K5QQCXC3AUUN6RQ, seg_pos={0 0}, upload=648645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ): protocol: expected piece hash; context canceled; EOF Jan 5 12:49:22.620775322 (741h20m22.491858357s) [noise=true , address=145.79.227.35:4485 , piece_id=PL2CJBTQPDRFKNHCTGZAVYYJJZNUI7INE7DS6K5QQCXC3AUUN6RQ, seg_pos={0 0}, upload=648645]: Long tail cancellation due to optimal threshold Jan 5 12:49:22.620812082 (741h20m22.491895157s) [noise=true , address=145.79.227.6:4285 , piece_id=JSDDDKI73UC44WITFP2FBUF4346FENQUOHGOFUPEMSHY577JB3AA, seg_pos={0 0}, upload=648645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Jan 5 12:49:22.620823563 (741h20m22.491906598s) [noise=true , address=145.79.227.6:4285 , piece_id=JSDDDKI73UC44WITFP2FBUF4346FENQUOHGOFUPEMSHY577JB3AA, seg_pos={0 0}, upload=648645]: Long tail cancellation due to optimal threshold Jan 5 12:49:22.620886423 (741h20m22.491969468s) [noise=true , address=77.90.31.15:4704 , piece_id=YZ3R4DWZXSDGWLEHSRN3EMV7FFOG7WN2TCKBUHYPC4M3FUMCRJGQ, seg_pos={0 0}, upload=648645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF Jan 5 12:49:22.620891513 (741h20m22.491974558s) [noise=true , address=77.90.31.15:4704 , piece_id=YZ3R4DWZXSDGWLEHSRN3EMV7FFOG7WN2TCKBUHYPC4M3FUMCRJGQ, seg_pos={0 0}, upload=648645]: Long tail cancellation due to optimal threshold Jan 5 12:49:22.620987486 (741h20m22.492070531s) [noise=true , address=145.79.227.34:4937 , piece_id=YF3SPF77UYHEWZCF5OSIHOOT3CLATLAGZUQFV6HHKFUJZGXO7XAA, seg_pos={0 0}, upload=648645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Jan 5 12:49:22.620992385 (741h20m22.49207542s) [noise=true , address=145.79.227.34:4937 , piece_id=YF3SPF77UYHEWZCF5OSIHOOT3CLATLAGZUQFV6HHKFUJZGXO7XAA, seg_pos={0 0}, upload=648645]: Long tail cancellation due to optimal threshold Jan 5 12:49:22.621007573 (741h20m22.492090618s) [seg_pos={0 0}, upload=648645]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 5 12:49:22.621077076 (741h20m22.492160111s) [upload=648645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:49:22.621081484 (741h20m22.492164529s) [upload=648645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:49:22.920850482 (741h20m22.791933527s) [upload=648645]: Done uploading segments... Jan 5 12:49:22.920877132 (741h20m22.791960197s) [upload=648645]: Upload finished. err: <nil> Duration: 43.42123554s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X............................X.] duration: 19.287066355s trace: https://jaeger.dev.storj.io/trace/53ac6558e03cb9af FAIL (2026-01-05 12:49:56 UTC) orbiter warning: context deadline exceeded Duration: 19.287117893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X...X..X.X..........X...XX...............XX.........X......] duration: 7.299249401s trace: https://jaeger.dev.storj.io/trace/28cbbf043d214b8f FAIL (2026-01-05 12:49:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5 12:45:21.016391497 (741h16m20.887474532s) [upload=648589]: Starting upload Jan 5 12:45:21.016449508 (741h16m20.887532543s) [upload=648589]: Done starting upload Jan 5 12:45:21.016606537 (741h16m20.887689572s) [upload=648589]: Uploading segments... Jan 5 12:45:21.016706537 (741h16m20.887789582s) [upload=648589]: Got next segment. Inline: false Jan 5 12:45:21.016739780 (741h16m20.887822825s) [upload=648589]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 5 12:45:21.016744811 (741h16m20.887827846s) [upload=648589]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 5 12:45:21.268595884 (741h16m21.139678929s) [seg_pos={0 0}, upload=648589]: Begin upload segment... Jan 5 12:45:21.268642443 (741h16m21.139725488s) [seg_pos={0 0}, upload=648589]: Done begin upload segment. Jan 5 12:45:21.268730540 (741h16m21.139813575s) [noise=true , address=145.79.227.22:4509 , piece_id=25JPBK3T7REYJP2WLO5K4LBLVIONOTRD5752G7MZWUHNXQ36562Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.270879172 (741h16m21.141962207s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.271442805 (741h16m21.14252584s) [noise=true , address=145.79.227.20:4437 , piece_id=VN5TKKWDYLGS7DBM5JKB6K2USRYFC7FUB4ZFHV3XJCJXZZ756G3Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.271905155 (741h16m21.14298819s) [noise=true , address=145.79.227.26:4653 , piece_id=DCMR6IJGZQELRMNI7LTMUPTCLSSL3U4JFR5TNZG2P36YSRN565TA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.272351776 (741h16m21.143434821s) [noise=true , address=145.79.227.48:4303 , piece_id=2D5IDHWWJXXZZH5WEL5UUDPWCCQJSYLB3HRWUHJAFHIJ6HQC7VZA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.272842941 (741h16m21.143925986s) [noise=true , address=145.79.227.28:4725 , piece_id=RHBNZJYMSWERVLZNUAEYSJNYCBWBKQZL72DX7AXGKU7V5PUAPM3Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.273284582 (741h16m21.144367627s) [noise=true , address=145.79.227.38:4979 , piece_id=BI5R7DQQFUDGR4AKJCXO6VUBKYKAZI5EAF5BWICVCDGD64NPQJQA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.273686899 (741h16m21.144769934s) [noise=true , address=45.91.116.14:4192 , piece_id=EUQFZAAJX5MN7TCT6U4K4VNL2UND56WESFXBDEL4K3FYBKZBNCNQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.274174016 (741h16m21.145257061s) [noise=true , address=145.79.227.50:4374 , piece_id=NFZSOJMZOLZKF7V52F37WAPSN6V7ZFYTPWY43ICQWIKDPPP4MHUA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.274699497 (741h16m21.145782532s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.275224597 (741h16m21.146307632s) [noise=true , address=145.79.227.5:4222 , piece_id=B2LPQWSTK6ESRTVV5LGUJZVXL277TV3FTCN6FFGXGVUGCRPBVYUA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.275731632 (741h16m21.146814677s) [noise=true , address=145.79.227.42:4121 , piece_id=RZNCW5NERG453IQLF564CPOUEEPLFK4GIZUWCV4WHHVZ5DIW66GQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.277290099 (741h16m21.148373134s) [noise=true , address=77.90.31.19:4842 , piece_id=F7E2LUJMCYFWHOGNSM2VXUTTMCFCP37VZQLCLP5VD2FTN5CQY7NQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.279405628 (741h16m21.150488673s) [noise=true , address=77.90.31.18:4808 , piece_id=VLTWI3TUQ5DZWJS7KYATLVDHP6RK3WPRY4I65KVAHJUPHBMOCY7Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.281446874 (741h16m21.152529919s) [noise=true , address=145.79.227.37:4565 , piece_id=VCJGFGJDQSLNSKMOAKDZZ5QFXVPA3DEWRCWFWRT6EIKRHZJZGR5A, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.283204051 (741h16m21.154287096s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:22.966583542 (741h16m22.837666587s) [upload=648589]: Next returned nil segment Jan 5 12:45:22.966601036 (741h16m22.837684081s) [upload=648589]: Waiting for error group managing segments... Jan 5 12:45:23.259998799 (741h16m23.131081834s) [noise=true , address=145.79.227.22:4509 , piece_id=25JPBK3T7REYJP2WLO5K4LBLVIONOTRD5752G7MZWUHNXQ36562Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.266672608 (741h16m23.137755653s) [noise=true , address=145.79.227.28:4725 , piece_id=RHBNZJYMSWERVLZNUAEYSJNYCBWBKQZL72DX7AXGKU7V5PUAPM3Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.268418462 (741h16m23.139501507s) [noise=true , address=145.79.227.5:4222 , piece_id=B2LPQWSTK6ESRTVV5LGUJZVXL277TV3FTCN6FFGXGVUGCRPBVYUA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.275051953 (741h16m23.146134998s) [noise=true , address=145.79.227.38:4979 , piece_id=BI5R7DQQFUDGR4AKJCXO6VUBKYKAZI5EAF5BWICVCDGD64NPQJQA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.277622617 (741h16m23.148705662s) [noise=true , address=145.79.227.37:4565 , piece_id=VCJGFGJDQSLNSKMOAKDZZ5QFXVPA3DEWRCWFWRT6EIKRHZJZGR5A, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.288675784 (741h16m23.159758819s) [noise=true , address=145.79.227.20:4437 , piece_id=VN5TKKWDYLGS7DBM5JKB6K2USRYFC7FUB4ZFHV3XJCJXZZ756G3Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.288731430 (741h16m23.159814475s) [noise=true , address=145.79.227.48:4303 , piece_id=2D5IDHWWJXXZZH5WEL5UUDPWCCQJSYLB3HRWUHJAFHIJ6HQC7VZA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.290197542 (741h16m23.161280587s) [noise=true , address=145.79.227.26:4653 , piece_id=DCMR6IJGZQELRMNI7LTMUPTCLSSL3U4JFR5TNZG2P36YSRN565TA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.291884975 (741h16m23.16296803s) [noise=true , address=145.79.227.50:4374 , piece_id=NFZSOJMZOLZKF7V52F37WAPSN6V7ZFYTPWY43ICQWIKDPPP4MHUA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.292911229 (741h16m23.163994274s) [noise=true , address=145.79.227.42:4121 , piece_id=RZNCW5NERG453IQLF564CPOUEEPLFK4GIZUWCV4WHHVZ5DIW66GQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.808597866 (741h16m23.679680911s) [noise=true , address=45.91.116.14:4192 , piece_id=EUQFZAAJX5MN7TCT6U4K4VNL2UND56WESFXBDEL4K3FYBKZBNCNQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:24.376048792 (741h16m24.247131837s) [noise=true , address=77.90.31.18:4808 , piece_id=VLTWI3TUQ5DZWJS7KYATLVDHP6RK3WPRY4I65KVAHJUPHBMOCY7Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:25.047325497 (741h16m24.918408542s) [noise=true , address=77.90.31.19:4842 , piece_id=F7E2LUJMCYFWHOGNSM2VXUTTMCFCP37VZQLCLP5VD2FTN5CQY7NQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:25.047352187 (741h16m24.918435222s) [seg_pos={0 0}, upload=648589]: Segment reached optimal threshold of 13 pieces. Jan 5 12:45:25.047587886 (741h16m24.918670931s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jan 5 12:45:25.047597243 (741h16m24.918680288s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047601892 (741h16m24.918684947s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection Jan 5 12:45:25.047610149 (741h16m24.918693194s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047827893 (741h16m24.918910938s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Jan 5 12:45:25.047837501 (741h16m24.918920546s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047856027 (741h16m24.918939072s) [seg_pos={0 0}, upload=648589]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 5 12:45:25.047959614 (741h16m24.919042659s) [upload=648589]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:45:25.047967589 (741h16m24.919050624s) [upload=648589]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:45:25.345497861 (741h16m25.216580906s) [upload=648589]: Done uploading segments... Jan 5 12:45:25.345532617 (741h16m25.216615662s) [upload=648589]: Upload finished. err: <nil> Duration: 7.299279307s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...............X....X.X.....X.....X..X..X...X....XX.X...XXX.] duration: 28.624972853s trace: https://jaeger.dev.storj.io/trace/16381dc260dc9d1a FAIL (2026-01-05 12:49:53 UTC) orbiter warning: context deadline exceeded Duration: 28.624995025s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 3.559688909s trace: https://jaeger.dev.storj.io/trace/2e8e8269d2d4542e FAIL (2026-01-05 12:49:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559722942s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X............................X] duration: 1m0.001566895s trace: https://jaeger.dev.storj.io/trace/53ac6558e03cb9af FAIL (2026-01-05 12:49:31 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001577836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [..X...XX.XX.X....X....X..XX..X.XXX.X.XXX....X....XX.....XX.X] duration: 1m0.00443463s trace: https://jaeger.dev.storj.io/trace/43207a9218e12518 FAIL (2026-01-05 12:49:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.249518773 (741h19m19.120601818s) [noise=true , address=145.79.227.40:4074 , piece_id=VVSMTFRMZURTI2HW6IRF3A2ZRRYC24DF2AJTQLJG3JT4OQYXN5LA, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.249948892 (741h19m19.121031947s) [noise=true , address=45.91.116.8:4061 , piece_id=FL2TAELE6CZTQVABMGOMDX6ZR4GRUPMQE7V37R5JLEIH7VPCY43A, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.250128875 (741h19m19.12121192s) [noise=true , address=45.91.116.4:4958 , piece_id=D6DPLSU2GN3TUXJVTMCD2NMOJLKKUDYRLZII4AM5PEYP63C4KSLQ, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.250525711 (741h19m19.121608746s) [noise=true , address=145.79.227.43:4168 , piece_id=JKMCC3GNQGD57NH4ANR2PLLJCCUOYGHFSHA73AYLKWQJHR3GF44A, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.250549185 (741h19m19.12163223s) [noise=true , address=64.49.13.155:4920 , piece_id=KC42X46GPC7EPCMLGWM7XZOLWYGKXR3GKCMDMAGGW7YUMWJXFONQ, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.250944739 (741h19m19.122027774s) [noise=true , address=145.79.227.20:4467 , piece_id=ID4ZHKJ7QEGCYW2RFBAGL2HSEHYUGYQGY7PJTNEJJV53BHKGJ4XA, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.251049037 (741h19m19.122132082s) [noise=true , address=145.79.227.11:4734 , piece_id=3YI2ZCDSV6BGY2UW7IWKS75WHWT3BOSJUQFAADA3BIVNRKV2W3IA, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.251338648 (741h19m19.122421684s) [noise=true , address=77.90.31.10:4529 , piece_id=WPSEZMEDB636FSAIXIG6GFNHQDWPRXQDUB4OOH6ACZ3OIUNC7KDA, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.251558206 (741h19m19.122641241s) [noise=true , address=45.91.116.13:4157 , piece_id=AFJNQVIO245DK6BOOFLBUWPJH4AXYU2DKFPWIGBWF5LONEWWFH4Q, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.251714865 (741h19m19.12279791s) [noise=true , address=193.149.164.143:4704 , piece_id=5GF3F2D4LFHGSWQYDKRLFZEBHJIXP7MHXNLNVNPA5F5QF4UTF7WA, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.252058740 (741h19m19.123141785s) [noise=true , address=145.79.227.13:4808 , piece_id=3JWDP6MDS5VFKDCQJ6E7TWDVJF63LFHAZB2EAM4HPMZE3522B6NA, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.252098455 (741h19m19.123181501s) [noise=true , address=145.79.227.33:4873 , piece_id=OGF6AIMMEW7UESAOKYJCWLUCIPXAMDSWSECGCISQWO3EJ7IP42QQ, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.252516021 (741h19m19.123599086s) [noise=true , address=145.79.227.30:4785 , piece_id=W7XIIG3JK5IQ2SY4ADLN2BP6ONL4YTG3K3ZJU62CM3AIXYMWXZRA, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.252583279 (741h19m19.123666314s) [noise=true , address=145.79.227.35:4490 , piece_id=WLKZDFVKIUXGI4O5NOYXBCLQ573MDDCQMK7RPZDTRJWB2AIGDGTA, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:19.252918597 (741h19m19.124001642s) [noise=true , address=64.49.13.169:4709 , piece_id=ESEN23INSXMZJ52QQR3IWTEICUAO5QVJIVVH5EXOC3LK24VQUK4Q, seg_pos={0 0}, upload=648629]: Uploading piece... Jan 5 12:48:20.703345022 (741h19m20.574428067s) [upload=648629]: Next returned nil segment Jan 5 12:48:20.703377153 (741h19m20.574460198s) [upload=648629]: Waiting for error group managing segments... Jan 5 12:48:21.085633258 (741h19m20.956716303s) [noise=true , address=64.49.13.155:4920 , piece_id=KC42X46GPC7EPCMLGWM7XZOLWYGKXR3GKCMDMAGGW7YUMWJXFONQ, seg_pos={0 0}, upload=648629]: Done uploading piece. err: <nil> Jan 5 12:48:21.119530444 (741h19m20.990613489s) [noise=true , address=145.79.227.11:4734 , piece_id=3YI2ZCDSV6BGY2UW7IWKS75WHWT3BOSJUQFAADA3BIVNRKV2W3IA, seg_pos={0 0}, upload=648629]: Done uploading piece. err: <nil> Jan 5 12:48:21.164215261 (741h19m21.035298316s) [noise=true , address=145.79.227.13:4808 , piece_id=3JWDP6MDS5VFKDCQJ6E7TWDVJF63LFHAZB2EAM4HPMZE3522B6NA, seg_pos={0 0}, upload=648629]: Done uploading piece. err: <nil> Jan 5 12:48:21.564172872 (741h19m21.435255927s) [noise=true , address=45.91.116.8:4061 , piece_id=FL2TAELE6CZTQVABMGOMDX6ZR4GRUPMQE7V37R5JLEIH7VPCY43A, seg_pos={0 0}, upload=648629]: Done uploading piece. err: <nil> Jan 5 12:48:21.570912545 (741h19m21.44199558s) [noise=true , address=145.79.227.20:4467 , piece_id=ID4ZHKJ7QEGCYW2RFBAGL2HSEHYUGYQGY7PJTNEJJV53BHKGJ4XA, seg_pos={0 0}, upload=648629]: Done uploading piece. err: <nil> Jan 5 12:48:21.602835221 (741h19m21.473918266s) [noise=true , address=45.91.116.4:4958 , piece_id=D6DPLSU2GN3TUXJVTMCD2NMOJLKKUDYRLZII4AM5PEYP63C4KSLQ, seg_pos={0 0}, upload=648629]: Done uploading piece. err: <nil> Jan 5 12:48:21.676617554 (741h19m21.547700589s) [noise=true , address=45.91.116.14:4210 , piece_id=RJBIKVN36HOXG6JMKWLXDDLESZVRUOUST2PLLV4EFEPNFFRI3I6A, seg_pos={0 0}, upload=648629]: Done uploading piece. err: <nil> Jan 5 12:48:21.803406029 (741h19m21.674489074s) [noise=true , address=77.90.31.10:4529 , piece_id=WPSEZMEDB636FSAIXIG6GFNHQDWPRXQDUB4OOH6ACZ3OIUNC7KDA, seg_pos={0 0}, upload=648629]: Done uploading piece. err: <nil> Jan 5 12:48:21.936113283 (741h19m21.807196328s) [noise=true , address=193.149.164.143:4704 , piece_id=5GF3F2D4LFHGSWQYDKRLFZEBHJIXP7MHXNLNVNPA5F5QF4UTF7WA, seg_pos={0 0}, upload=648629]: Done uploading piece. err: <nil> Jan 5 12:48:22.008083126 (741h19m21.879166171s) [noise=true , address=145.79.227.30:4785 , piece_id=W7XIIG3JK5IQ2SY4ADLN2BP6ONL4YTG3K3ZJU62CM3AIXYMWXZRA, seg_pos={0 0}, upload=648629]: Done uploading piece. err: <nil> Jan 5 12:48:22.017432558 (741h19m21.888515593s) [noise=true , address=145.79.227.43:4168 , piece_id=JKMCC3GNQGD57NH4ANR2PLLJCCUOYGHFSHA73AYLKWQJHR3GF44A, seg_pos={0 0}, upload=648629]: Done uploading piece. err: <nil> Jan 5 12:48:22.053404885 (741h19m21.92448793s) [noise=true , address=145.79.227.44:4209 , piece_id=LHK2J4TYPZ76FMCA5MKIHPVAVZNRMSM4OQZARGNW56GEIQGHB2PA, seg_pos={0 0}, upload=648629]: Done uploading piece. err: <nil> Jan 5 12:48:22.076056938 (741h19m21.947139983s) [noise=true , address=145.79.227.34:4925 , piece_id=PYK6DHL7BLEE5BRR45YSCIPSG7XMVSIW4SPTAFHT5NXXKUM2FKTQ, seg_pos={0 0}, upload=648629]: Done uploading piece. err: <nil> Jan 5 12:48:22.094724464 (741h19m21.965807499s) [noise=true , address=145.79.227.40:4074 , piece_id=VVSMTFRMZURTI2HW6IRF3A2ZRRYC24DF2AJTQLJG3JT4OQYXN5LA, seg_pos={0 0}, upload=648629]: Done uploading piece. err: <nil> Jan 5 12:48:22.094750023 (741h19m21.965833068s) [seg_pos={0 0}, upload=648629]: Segment reached optimal threshold of 14 pieces. Jan 5 12:48:22.095080391 (741h19m21.966163456s) [noise=true , address=145.79.227.35:4490 , piece_id=WLKZDFVKIUXGI4O5NOYXBCLQ573MDDCQMK7RPZDTRJWB2AIGDGTA, seg_pos={0 0}, upload=648629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Jan 5 12:48:22.095090951 (741h19m21.966174007s) [noise=true , address=145.79.227.35:4490 , piece_id=WLKZDFVKIUXGI4O5NOYXBCLQ573MDDCQMK7RPZDTRJWB2AIGDGTA, seg_pos={0 0}, upload=648629]: Long tail cancellation due to optimal threshold Jan 5 12:48:22.095373710 (741h19m21.966456745s) [noise=true , address=64.49.13.169:4709 , piece_id=ESEN23INSXMZJ52QQR3IWTEICUAO5QVJIVVH5EXOC3LK24VQUK4Q, seg_pos={0 0}, upload=648629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:60842->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:60842->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60842->64.49.13.169:4709: use of closed network connection Jan 5 12:48:22.095381605 (741h19m21.96646465s) [noise=true , address=64.49.13.169:4709 , piece_id=ESEN23INSXMZJ52QQR3IWTEICUAO5QVJIVVH5EXOC3LK24VQUK4Q, seg_pos={0 0}, upload=648629]: Long tail cancellation due to optimal threshold Jan 5 12:48:22.095580404 (741h19m21.966663439s) [noise=true , address=145.79.227.33:4873 , piece_id=OGF6AIMMEW7UESAOKYJCWLUCIPXAMDSWSECGCISQWO3EJ7IP42QQ, seg_pos={0 0}, upload=648629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Jan 5 12:48:22.095587978 (741h19m21.966671013s) [noise=true , address=145.79.227.33:4873 , piece_id=OGF6AIMMEW7UESAOKYJCWLUCIPXAMDSWSECGCISQWO3EJ7IP42QQ, seg_pos={0 0}, upload=648629]: Long tail cancellation due to optimal threshold Jan 5 12:48:22.095788550 (741h19m21.966871615s) [noise=true , address=45.91.116.13:4157 , piece_id=AFJNQVIO245DK6BOOFLBUWPJH4AXYU2DKFPWIGBWF5LONEWWFH4Q, seg_pos={0 0}, upload=648629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj): protocol: read tcp 188.166.225.170:33414->45.91.116.13:4157: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 5 12:48:22.095799781 (741h19m21.966882827s) [noise=true , address=45.91.116.13:4157 , piece_id=AFJNQVIO245DK6BOOFLBUWPJH4AXYU2DKFPWIGBWF5LONEWWFH4Q, seg_pos={0 0}, upload=648629]: Long tail cancellation due to optimal threshold Jan 5 12:48:22.095823557 (741h19m21.966906602s) [seg_pos={0 0}, upload=648629]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 5 12:48:22.095971508 (741h19m21.967054554s) [upload=648629]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:48:22.095978952 (741h19m21.967062028s) [upload=648629]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:48:22.378623291 (741h19m22.249706336s) [upload=648629]: Done uploading segments... Jan 5 12:48:22.378647627 (741h19m22.249730672s) [upload=648629]: Upload finished. err: <nil> Duration: 1m0.004457383s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.002279221s trace: https://jaeger.dev.storj.io/trace/2e8e8269d2d4542e FAIL (2026-01-05 12:49:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0023051s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................X.............X.......................] duration: 10.205619283s trace: https://jaeger.dev.storj.io/trace/5f0936145b5399cd FAIL (2026-01-05 12:49:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 10.205644179s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.............X......................] duration: 15.370253404s trace: https://jaeger.dev.storj.io/trace/5f0936145b5399cd FAIL (2026-01-05 12:48:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 15.370269184s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [.XXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001143808s trace: https://jaeger.dev.storj.io/trace/4a770961ed63a12b FAIL (2026-01-05 12:48:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001157054s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [................X....X.X.....X.....X..X..X...X....XX.X...XXX] duration: 30.00076316s trace: https://jaeger.dev.storj.io/trace/16381dc260dc9d1a FAIL (2026-01-05 12:48:48 UTC) orbiter warning: context deadline exceeded Duration: 30.000786916s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X...X..X.X..........X...XX...............XX.........X.....] duration: 5.864127064s trace: https://jaeger.dev.storj.io/trace/28cbbf043d214b8f FAIL (2026-01-05 12:48:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5 12:45:21.016391497 (741h16m20.887474532s) [upload=648589]: Starting upload Jan 5 12:45:21.016449508 (741h16m20.887532543s) [upload=648589]: Done starting upload Jan 5 12:45:21.016606537 (741h16m20.887689572s) [upload=648589]: Uploading segments... Jan 5 12:45:21.016706537 (741h16m20.887789582s) [upload=648589]: Got next segment. Inline: false Jan 5 12:45:21.016739780 (741h16m20.887822825s) [upload=648589]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 5 12:45:21.016744811 (741h16m20.887827846s) [upload=648589]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 5 12:45:21.268595884 (741h16m21.139678929s) [seg_pos={0 0}, upload=648589]: Begin upload segment... Jan 5 12:45:21.268642443 (741h16m21.139725488s) [seg_pos={0 0}, upload=648589]: Done begin upload segment. Jan 5 12:45:21.268730540 (741h16m21.139813575s) [noise=true , address=145.79.227.22:4509 , piece_id=25JPBK3T7REYJP2WLO5K4LBLVIONOTRD5752G7MZWUHNXQ36562Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.270879172 (741h16m21.141962207s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.271442805 (741h16m21.14252584s) [noise=true , address=145.79.227.20:4437 , piece_id=VN5TKKWDYLGS7DBM5JKB6K2USRYFC7FUB4ZFHV3XJCJXZZ756G3Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.271905155 (741h16m21.14298819s) [noise=true , address=145.79.227.26:4653 , piece_id=DCMR6IJGZQELRMNI7LTMUPTCLSSL3U4JFR5TNZG2P36YSRN565TA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.272351776 (741h16m21.143434821s) [noise=true , address=145.79.227.48:4303 , piece_id=2D5IDHWWJXXZZH5WEL5UUDPWCCQJSYLB3HRWUHJAFHIJ6HQC7VZA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.272842941 (741h16m21.143925986s) [noise=true , address=145.79.227.28:4725 , piece_id=RHBNZJYMSWERVLZNUAEYSJNYCBWBKQZL72DX7AXGKU7V5PUAPM3Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.273284582 (741h16m21.144367627s) [noise=true , address=145.79.227.38:4979 , piece_id=BI5R7DQQFUDGR4AKJCXO6VUBKYKAZI5EAF5BWICVCDGD64NPQJQA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.273686899 (741h16m21.144769934s) [noise=true , address=45.91.116.14:4192 , piece_id=EUQFZAAJX5MN7TCT6U4K4VNL2UND56WESFXBDEL4K3FYBKZBNCNQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.274174016 (741h16m21.145257061s) [noise=true , address=145.79.227.50:4374 , piece_id=NFZSOJMZOLZKF7V52F37WAPSN6V7ZFYTPWY43ICQWIKDPPP4MHUA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.274699497 (741h16m21.145782532s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.275224597 (741h16m21.146307632s) [noise=true , address=145.79.227.5:4222 , piece_id=B2LPQWSTK6ESRTVV5LGUJZVXL277TV3FTCN6FFGXGVUGCRPBVYUA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.275731632 (741h16m21.146814677s) [noise=true , address=145.79.227.42:4121 , piece_id=RZNCW5NERG453IQLF564CPOUEEPLFK4GIZUWCV4WHHVZ5DIW66GQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.277290099 (741h16m21.148373134s) [noise=true , address=77.90.31.19:4842 , piece_id=F7E2LUJMCYFWHOGNSM2VXUTTMCFCP37VZQLCLP5VD2FTN5CQY7NQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.279405628 (741h16m21.150488673s) [noise=true , address=77.90.31.18:4808 , piece_id=VLTWI3TUQ5DZWJS7KYATLVDHP6RK3WPRY4I65KVAHJUPHBMOCY7Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.281446874 (741h16m21.152529919s) [noise=true , address=145.79.227.37:4565 , piece_id=VCJGFGJDQSLNSKMOAKDZZ5QFXVPA3DEWRCWFWRT6EIKRHZJZGR5A, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.283204051 (741h16m21.154287096s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:22.966583542 (741h16m22.837666587s) [upload=648589]: Next returned nil segment Jan 5 12:45:22.966601036 (741h16m22.837684081s) [upload=648589]: Waiting for error group managing segments... Jan 5 12:45:23.259998799 (741h16m23.131081834s) [noise=true , address=145.79.227.22:4509 , piece_id=25JPBK3T7REYJP2WLO5K4LBLVIONOTRD5752G7MZWUHNXQ36562Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.266672608 (741h16m23.137755653s) [noise=true , address=145.79.227.28:4725 , piece_id=RHBNZJYMSWERVLZNUAEYSJNYCBWBKQZL72DX7AXGKU7V5PUAPM3Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.268418462 (741h16m23.139501507s) [noise=true , address=145.79.227.5:4222 , piece_id=B2LPQWSTK6ESRTVV5LGUJZVXL277TV3FTCN6FFGXGVUGCRPBVYUA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.275051953 (741h16m23.146134998s) [noise=true , address=145.79.227.38:4979 , piece_id=BI5R7DQQFUDGR4AKJCXO6VUBKYKAZI5EAF5BWICVCDGD64NPQJQA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.277622617 (741h16m23.148705662s) [noise=true , address=145.79.227.37:4565 , piece_id=VCJGFGJDQSLNSKMOAKDZZ5QFXVPA3DEWRCWFWRT6EIKRHZJZGR5A, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.288675784 (741h16m23.159758819s) [noise=true , address=145.79.227.20:4437 , piece_id=VN5TKKWDYLGS7DBM5JKB6K2USRYFC7FUB4ZFHV3XJCJXZZ756G3Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.288731430 (741h16m23.159814475s) [noise=true , address=145.79.227.48:4303 , piece_id=2D5IDHWWJXXZZH5WEL5UUDPWCCQJSYLB3HRWUHJAFHIJ6HQC7VZA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.290197542 (741h16m23.161280587s) [noise=true , address=145.79.227.26:4653 , piece_id=DCMR6IJGZQELRMNI7LTMUPTCLSSL3U4JFR5TNZG2P36YSRN565TA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.291884975 (741h16m23.16296803s) [noise=true , address=145.79.227.50:4374 , piece_id=NFZSOJMZOLZKF7V52F37WAPSN6V7ZFYTPWY43ICQWIKDPPP4MHUA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.292911229 (741h16m23.163994274s) [noise=true , address=145.79.227.42:4121 , piece_id=RZNCW5NERG453IQLF564CPOUEEPLFK4GIZUWCV4WHHVZ5DIW66GQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.808597866 (741h16m23.679680911s) [noise=true , address=45.91.116.14:4192 , piece_id=EUQFZAAJX5MN7TCT6U4K4VNL2UND56WESFXBDEL4K3FYBKZBNCNQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:24.376048792 (741h16m24.247131837s) [noise=true , address=77.90.31.18:4808 , piece_id=VLTWI3TUQ5DZWJS7KYATLVDHP6RK3WPRY4I65KVAHJUPHBMOCY7Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:25.047325497 (741h16m24.918408542s) [noise=true , address=77.90.31.19:4842 , piece_id=F7E2LUJMCYFWHOGNSM2VXUTTMCFCP37VZQLCLP5VD2FTN5CQY7NQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:25.047352187 (741h16m24.918435222s) [seg_pos={0 0}, upload=648589]: Segment reached optimal threshold of 13 pieces. Jan 5 12:45:25.047587886 (741h16m24.918670931s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jan 5 12:45:25.047597243 (741h16m24.918680288s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047601892 (741h16m24.918684947s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection Jan 5 12:45:25.047610149 (741h16m24.918693194s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047827893 (741h16m24.918910938s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Jan 5 12:45:25.047837501 (741h16m24.918920546s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047856027 (741h16m24.918939072s) [seg_pos={0 0}, upload=648589]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 5 12:45:25.047959614 (741h16m24.919042659s) [upload=648589]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:45:25.047967589 (741h16m24.919050624s) [upload=648589]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:45:25.345497861 (741h16m25.216580906s) [upload=648589]: Done uploading segments... Jan 5 12:45:25.345532617 (741h16m25.216615662s) [upload=648589]: Upload finished. err: <nil> Duration: 5.864158944s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 41.12830582s trace: https://jaeger.dev.storj.io/trace/179e4cdf9f7972e0 FAIL (2026-01-05 12:48:31 UTC) orbiter warning: context deadline exceeded Duration: 41.128333484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..X...X..X.X..........X...XX...............XX.........X....] duration: 5.77294661s trace: https://jaeger.dev.storj.io/trace/28cbbf043d214b8f FAIL (2026-01-05 12:48:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5 12:45:21.016391497 (741h16m20.887474532s) [upload=648589]: Starting upload Jan 5 12:45:21.016449508 (741h16m20.887532543s) [upload=648589]: Done starting upload Jan 5 12:45:21.016606537 (741h16m20.887689572s) [upload=648589]: Uploading segments... Jan 5 12:45:21.016706537 (741h16m20.887789582s) [upload=648589]: Got next segment. Inline: false Jan 5 12:45:21.016739780 (741h16m20.887822825s) [upload=648589]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 5 12:45:21.016744811 (741h16m20.887827846s) [upload=648589]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 5 12:45:21.268595884 (741h16m21.139678929s) [seg_pos={0 0}, upload=648589]: Begin upload segment... Jan 5 12:45:21.268642443 (741h16m21.139725488s) [seg_pos={0 0}, upload=648589]: Done begin upload segment. Jan 5 12:45:21.268730540 (741h16m21.139813575s) [noise=true , address=145.79.227.22:4509 , piece_id=25JPBK3T7REYJP2WLO5K4LBLVIONOTRD5752G7MZWUHNXQ36562Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.270879172 (741h16m21.141962207s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.271442805 (741h16m21.14252584s) [noise=true , address=145.79.227.20:4437 , piece_id=VN5TKKWDYLGS7DBM5JKB6K2USRYFC7FUB4ZFHV3XJCJXZZ756G3Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.271905155 (741h16m21.14298819s) [noise=true , address=145.79.227.26:4653 , piece_id=DCMR6IJGZQELRMNI7LTMUPTCLSSL3U4JFR5TNZG2P36YSRN565TA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.272351776 (741h16m21.143434821s) [noise=true , address=145.79.227.48:4303 , piece_id=2D5IDHWWJXXZZH5WEL5UUDPWCCQJSYLB3HRWUHJAFHIJ6HQC7VZA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.272842941 (741h16m21.143925986s) [noise=true , address=145.79.227.28:4725 , piece_id=RHBNZJYMSWERVLZNUAEYSJNYCBWBKQZL72DX7AXGKU7V5PUAPM3Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.273284582 (741h16m21.144367627s) [noise=true , address=145.79.227.38:4979 , piece_id=BI5R7DQQFUDGR4AKJCXO6VUBKYKAZI5EAF5BWICVCDGD64NPQJQA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.273686899 (741h16m21.144769934s) [noise=true , address=45.91.116.14:4192 , piece_id=EUQFZAAJX5MN7TCT6U4K4VNL2UND56WESFXBDEL4K3FYBKZBNCNQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.274174016 (741h16m21.145257061s) [noise=true , address=145.79.227.50:4374 , piece_id=NFZSOJMZOLZKF7V52F37WAPSN6V7ZFYTPWY43ICQWIKDPPP4MHUA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.274699497 (741h16m21.145782532s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.275224597 (741h16m21.146307632s) [noise=true , address=145.79.227.5:4222 , piece_id=B2LPQWSTK6ESRTVV5LGUJZVXL277TV3FTCN6FFGXGVUGCRPBVYUA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.275731632 (741h16m21.146814677s) [noise=true , address=145.79.227.42:4121 , piece_id=RZNCW5NERG453IQLF564CPOUEEPLFK4GIZUWCV4WHHVZ5DIW66GQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.277290099 (741h16m21.148373134s) [noise=true , address=77.90.31.19:4842 , piece_id=F7E2LUJMCYFWHOGNSM2VXUTTMCFCP37VZQLCLP5VD2FTN5CQY7NQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.279405628 (741h16m21.150488673s) [noise=true , address=77.90.31.18:4808 , piece_id=VLTWI3TUQ5DZWJS7KYATLVDHP6RK3WPRY4I65KVAHJUPHBMOCY7Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.281446874 (741h16m21.152529919s) [noise=true , address=145.79.227.37:4565 , piece_id=VCJGFGJDQSLNSKMOAKDZZ5QFXVPA3DEWRCWFWRT6EIKRHZJZGR5A, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.283204051 (741h16m21.154287096s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:22.966583542 (741h16m22.837666587s) [upload=648589]: Next returned nil segment Jan 5 12:45:22.966601036 (741h16m22.837684081s) [upload=648589]: Waiting for error group managing segments... Jan 5 12:45:23.259998799 (741h16m23.131081834s) [noise=true , address=145.79.227.22:4509 , piece_id=25JPBK3T7REYJP2WLO5K4LBLVIONOTRD5752G7MZWUHNXQ36562Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.266672608 (741h16m23.137755653s) [noise=true , address=145.79.227.28:4725 , piece_id=RHBNZJYMSWERVLZNUAEYSJNYCBWBKQZL72DX7AXGKU7V5PUAPM3Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.268418462 (741h16m23.139501507s) [noise=true , address=145.79.227.5:4222 , piece_id=B2LPQWSTK6ESRTVV5LGUJZVXL277TV3FTCN6FFGXGVUGCRPBVYUA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.275051953 (741h16m23.146134998s) [noise=true , address=145.79.227.38:4979 , piece_id=BI5R7DQQFUDGR4AKJCXO6VUBKYKAZI5EAF5BWICVCDGD64NPQJQA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.277622617 (741h16m23.148705662s) [noise=true , address=145.79.227.37:4565 , piece_id=VCJGFGJDQSLNSKMOAKDZZ5QFXVPA3DEWRCWFWRT6EIKRHZJZGR5A, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.288675784 (741h16m23.159758819s) [noise=true , address=145.79.227.20:4437 , piece_id=VN5TKKWDYLGS7DBM5JKB6K2USRYFC7FUB4ZFHV3XJCJXZZ756G3Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.288731430 (741h16m23.159814475s) [noise=true , address=145.79.227.48:4303 , piece_id=2D5IDHWWJXXZZH5WEL5UUDPWCCQJSYLB3HRWUHJAFHIJ6HQC7VZA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.290197542 (741h16m23.161280587s) [noise=true , address=145.79.227.26:4653 , piece_id=DCMR6IJGZQELRMNI7LTMUPTCLSSL3U4JFR5TNZG2P36YSRN565TA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.291884975 (741h16m23.16296803s) [noise=true , address=145.79.227.50:4374 , piece_id=NFZSOJMZOLZKF7V52F37WAPSN6V7ZFYTPWY43ICQWIKDPPP4MHUA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.292911229 (741h16m23.163994274s) [noise=true , address=145.79.227.42:4121 , piece_id=RZNCW5NERG453IQLF564CPOUEEPLFK4GIZUWCV4WHHVZ5DIW66GQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.808597866 (741h16m23.679680911s) [noise=true , address=45.91.116.14:4192 , piece_id=EUQFZAAJX5MN7TCT6U4K4VNL2UND56WESFXBDEL4K3FYBKZBNCNQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:24.376048792 (741h16m24.247131837s) [noise=true , address=77.90.31.18:4808 , piece_id=VLTWI3TUQ5DZWJS7KYATLVDHP6RK3WPRY4I65KVAHJUPHBMOCY7Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:25.047325497 (741h16m24.918408542s) [noise=true , address=77.90.31.19:4842 , piece_id=F7E2LUJMCYFWHOGNSM2VXUTTMCFCP37VZQLCLP5VD2FTN5CQY7NQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:25.047352187 (741h16m24.918435222s) [seg_pos={0 0}, upload=648589]: Segment reached optimal threshold of 13 pieces. Jan 5 12:45:25.047587886 (741h16m24.918670931s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jan 5 12:45:25.047597243 (741h16m24.918680288s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047601892 (741h16m24.918684947s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection Jan 5 12:45:25.047610149 (741h16m24.918693194s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047827893 (741h16m24.918910938s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Jan 5 12:45:25.047837501 (741h16m24.918920546s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047856027 (741h16m24.918939072s) [seg_pos={0 0}, upload=648589]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 5 12:45:25.047959614 (741h16m24.919042659s) [upload=648589]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:45:25.047967589 (741h16m24.919050624s) [upload=648589]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:45:25.345497861 (741h16m25.216580906s) [upload=648589]: Done uploading segments... Jan 5 12:45:25.345532617 (741h16m25.216615662s) [upload=648589]: Upload finished. err: <nil> Duration: 5.772976175s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................X.............X.....................] duration: 12.556054071s trace: https://jaeger.dev.storj.io/trace/5f0936145b5399cd FAIL (2026-01-05 12:48:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 12.556083537s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X..X...X..X.X..........X...XX...............XX.........X...] duration: 14.459481979s trace: https://jaeger.dev.storj.io/trace/28cbbf043d214b8f FAIL (2026-01-05 12:48:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5 12:45:21.016391497 (741h16m20.887474532s) [upload=648589]: Starting upload Jan 5 12:45:21.016449508 (741h16m20.887532543s) [upload=648589]: Done starting upload Jan 5 12:45:21.016606537 (741h16m20.887689572s) [upload=648589]: Uploading segments... Jan 5 12:45:21.016706537 (741h16m20.887789582s) [upload=648589]: Got next segment. Inline: false Jan 5 12:45:21.016739780 (741h16m20.887822825s) [upload=648589]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 5 12:45:21.016744811 (741h16m20.887827846s) [upload=648589]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 5 12:45:21.268595884 (741h16m21.139678929s) [seg_pos={0 0}, upload=648589]: Begin upload segment... Jan 5 12:45:21.268642443 (741h16m21.139725488s) [seg_pos={0 0}, upload=648589]: Done begin upload segment. Jan 5 12:45:21.268730540 (741h16m21.139813575s) [noise=true , address=145.79.227.22:4509 , piece_id=25JPBK3T7REYJP2WLO5K4LBLVIONOTRD5752G7MZWUHNXQ36562Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.270879172 (741h16m21.141962207s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.271442805 (741h16m21.14252584s) [noise=true , address=145.79.227.20:4437 , piece_id=VN5TKKWDYLGS7DBM5JKB6K2USRYFC7FUB4ZFHV3XJCJXZZ756G3Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.271905155 (741h16m21.14298819s) [noise=true , address=145.79.227.26:4653 , piece_id=DCMR6IJGZQELRMNI7LTMUPTCLSSL3U4JFR5TNZG2P36YSRN565TA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.272351776 (741h16m21.143434821s) [noise=true , address=145.79.227.48:4303 , piece_id=2D5IDHWWJXXZZH5WEL5UUDPWCCQJSYLB3HRWUHJAFHIJ6HQC7VZA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.272842941 (741h16m21.143925986s) [noise=true , address=145.79.227.28:4725 , piece_id=RHBNZJYMSWERVLZNUAEYSJNYCBWBKQZL72DX7AXGKU7V5PUAPM3Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.273284582 (741h16m21.144367627s) [noise=true , address=145.79.227.38:4979 , piece_id=BI5R7DQQFUDGR4AKJCXO6VUBKYKAZI5EAF5BWICVCDGD64NPQJQA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.273686899 (741h16m21.144769934s) [noise=true , address=45.91.116.14:4192 , piece_id=EUQFZAAJX5MN7TCT6U4K4VNL2UND56WESFXBDEL4K3FYBKZBNCNQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.274174016 (741h16m21.145257061s) [noise=true , address=145.79.227.50:4374 , piece_id=NFZSOJMZOLZKF7V52F37WAPSN6V7ZFYTPWY43ICQWIKDPPP4MHUA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.274699497 (741h16m21.145782532s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.275224597 (741h16m21.146307632s) [noise=true , address=145.79.227.5:4222 , piece_id=B2LPQWSTK6ESRTVV5LGUJZVXL277TV3FTCN6FFGXGVUGCRPBVYUA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.275731632 (741h16m21.146814677s) [noise=true , address=145.79.227.42:4121 , piece_id=RZNCW5NERG453IQLF564CPOUEEPLFK4GIZUWCV4WHHVZ5DIW66GQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.277290099 (741h16m21.148373134s) [noise=true , address=77.90.31.19:4842 , piece_id=F7E2LUJMCYFWHOGNSM2VXUTTMCFCP37VZQLCLP5VD2FTN5CQY7NQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.279405628 (741h16m21.150488673s) [noise=true , address=77.90.31.18:4808 , piece_id=VLTWI3TUQ5DZWJS7KYATLVDHP6RK3WPRY4I65KVAHJUPHBMOCY7Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.281446874 (741h16m21.152529919s) [noise=true , address=145.79.227.37:4565 , piece_id=VCJGFGJDQSLNSKMOAKDZZ5QFXVPA3DEWRCWFWRT6EIKRHZJZGR5A, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.283204051 (741h16m21.154287096s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:22.966583542 (741h16m22.837666587s) [upload=648589]: Next returned nil segment Jan 5 12:45:22.966601036 (741h16m22.837684081s) [upload=648589]: Waiting for error group managing segments... Jan 5 12:45:23.259998799 (741h16m23.131081834s) [noise=true , address=145.79.227.22:4509 , piece_id=25JPBK3T7REYJP2WLO5K4LBLVIONOTRD5752G7MZWUHNXQ36562Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.266672608 (741h16m23.137755653s) [noise=true , address=145.79.227.28:4725 , piece_id=RHBNZJYMSWERVLZNUAEYSJNYCBWBKQZL72DX7AXGKU7V5PUAPM3Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.268418462 (741h16m23.139501507s) [noise=true , address=145.79.227.5:4222 , piece_id=B2LPQWSTK6ESRTVV5LGUJZVXL277TV3FTCN6FFGXGVUGCRPBVYUA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.275051953 (741h16m23.146134998s) [noise=true , address=145.79.227.38:4979 , piece_id=BI5R7DQQFUDGR4AKJCXO6VUBKYKAZI5EAF5BWICVCDGD64NPQJQA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.277622617 (741h16m23.148705662s) [noise=true , address=145.79.227.37:4565 , piece_id=VCJGFGJDQSLNSKMOAKDZZ5QFXVPA3DEWRCWFWRT6EIKRHZJZGR5A, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.288675784 (741h16m23.159758819s) [noise=true , address=145.79.227.20:4437 , piece_id=VN5TKKWDYLGS7DBM5JKB6K2USRYFC7FUB4ZFHV3XJCJXZZ756G3Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.288731430 (741h16m23.159814475s) [noise=true , address=145.79.227.48:4303 , piece_id=2D5IDHWWJXXZZH5WEL5UUDPWCCQJSYLB3HRWUHJAFHIJ6HQC7VZA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.290197542 (741h16m23.161280587s) [noise=true , address=145.79.227.26:4653 , piece_id=DCMR6IJGZQELRMNI7LTMUPTCLSSL3U4JFR5TNZG2P36YSRN565TA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.291884975 (741h16m23.16296803s) [noise=true , address=145.79.227.50:4374 , piece_id=NFZSOJMZOLZKF7V52F37WAPSN6V7ZFYTPWY43ICQWIKDPPP4MHUA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.292911229 (741h16m23.163994274s) [noise=true , address=145.79.227.42:4121 , piece_id=RZNCW5NERG453IQLF564CPOUEEPLFK4GIZUWCV4WHHVZ5DIW66GQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.808597866 (741h16m23.679680911s) [noise=true , address=45.91.116.14:4192 , piece_id=EUQFZAAJX5MN7TCT6U4K4VNL2UND56WESFXBDEL4K3FYBKZBNCNQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:24.376048792 (741h16m24.247131837s) [noise=true , address=77.90.31.18:4808 , piece_id=VLTWI3TUQ5DZWJS7KYATLVDHP6RK3WPRY4I65KVAHJUPHBMOCY7Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:25.047325497 (741h16m24.918408542s) [noise=true , address=77.90.31.19:4842 , piece_id=F7E2LUJMCYFWHOGNSM2VXUTTMCFCP37VZQLCLP5VD2FTN5CQY7NQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:25.047352187 (741h16m24.918435222s) [seg_pos={0 0}, upload=648589]: Segment reached optimal threshold of 13 pieces. Jan 5 12:45:25.047587886 (741h16m24.918670931s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jan 5 12:45:25.047597243 (741h16m24.918680288s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047601892 (741h16m24.918684947s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection Jan 5 12:45:25.047610149 (741h16m24.918693194s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047827893 (741h16m24.918910938s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Jan 5 12:45:25.047837501 (741h16m24.918920546s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047856027 (741h16m24.918939072s) [seg_pos={0 0}, upload=648589]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 5 12:45:25.047959614 (741h16m24.919042659s) [upload=648589]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:45:25.047967589 (741h16m24.919050624s) [upload=648589]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:45:25.345497861 (741h16m25.216580906s) [upload=648589]: Done uploading segments... Jan 5 12:45:25.345532617 (741h16m25.216615662s) [upload=648589]: Upload finished. err: <nil> Duration: 14.459589184s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.................X....X.X.....X.....X..X..X...X....XX.X...XX] duration: 30.001146367s trace: https://jaeger.dev.storj.io/trace/36eca6eaa87ea63e FAIL (2026-01-05 12:47:54 UTC) orbiter warning: context deadline exceeded Duration: 30.001175442s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 50.438346281s trace: https://jaeger.dev.storj.io/trace/179e4cdf9f7972e0 FAIL (2026-01-05 12:47:46 UTC) orbiter warning: context deadline exceeded Duration: 50.43838835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X...XX.XX.X....X....X..XX..X.XXX.X.XXX....X....XX.....XX.] duration: 22.950918787s trace: https://jaeger.dev.storj.io/trace/2b9f2b9d1b3af09b FAIL (2026-01-05 12:47:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 116.11:4149 , piece_id=USMZ7R7R2D7AKO2NBNL4ANKTMD7YCFRCGMAJC4I3Q57WTVTP6TOA, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:52.454300627 (741h17m52.325383662s) [noise=true , address=145.79.227.32:4867 , piece_id=YYMB4ONF4FKGKRUKJ57W6FC57JOPKAI6QNPYBXGWKGF2YPYHL63A, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:52.454914587 (741h17m52.325997622s) [noise=true , address=145.79.227.29:4971 , piece_id=WOJZQ66XSFE4XRL2F3P5DAFN5BJP4JUYCLP6JDVXCMC4RRF77EHA, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:52.455129155 (741h17m52.32621219s) [noise=true , address=193.149.164.143:4696 , piece_id=7CTMW4C6NXQFOL45UVLL6B27JZC6PXUUKBCIIGRDUH6BROHSMDGA, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:52.455308757 (741h17m52.326391802s) [noise=true , address=145.79.227.40:4065 , piece_id=6O3F5YLQGESHONSZLQESSOVAIVIMWBOOSB7O6DSCDVWINNIK3DRA, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:52.455458331 (741h17m52.326541366s) [noise=true , address=77.90.31.19:4846 , piece_id=2HXBK56ZHRTWPRUDJ22SIBJJX4KKTWLJLQ4GRJCI7S7RPC25SYRA, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:52.455678792 (741h17m52.326761837s) [noise=true , address=145.79.227.30:4793 , piece_id=XQZNHYAYWEKN2K464RPFSE5ERXDEZUWXBRYBAVDLRWYPBKRBLSAQ, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:52.456036102 (741h17m52.327119137s) [noise=true , address=145.79.227.5:4239 , piece_id=FE4BDHDHQCCLJFGAIYS6SF47CQO353JRLHJVXBKR5K7YXWDSYCVA, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:52.456407139 (741h17m52.327490174s) [noise=true , address=77.90.31.10:4538 , piece_id=N23Q4IRVBYGEO6QJMUPJPKAB5MYWRB2ZVTPGTZZANUHHMWTOQP2A, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:52.456784327 (741h17m52.327867372s) [noise=true , address=45.91.116.8:4052 , piece_id=PHL2GIZ4KSJKRC3ZIBRSMFZ7AGCRPJ5YS4P3IUDJQ6IO35CZFK6Q, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:52.457305780 (741h17m52.328388815s) [noise=true , address=145.79.227.43:4182 , piece_id=2NTCCJ4CTQXAZB7FEVMFLIZG3IBWBFB5AHOVIJVWZBZADY6NIK6Q, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:52.457585162 (741h17m52.328668197s) [noise=true , address=64.49.13.169:4708 , piece_id=2B5EKXGQZU4B2AUV63P6IRTT3EN3EERCZTSII5HYIYLCWL5YE3AQ, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:52.458903662 (741h17m52.329986697s) [noise=true , address=145.79.227.44:4226 , piece_id=KYYYQE2OMDUBDE6E7NC66AXCQZRG5XRETSRFC5EBXR5YJBOT3OUQ, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:52.459382404 (741h17m52.330465439s) [noise=true , address=145.79.227.35:4509 , piece_id=YYTKU2E7L2YDPXOFNM2WKCAZ6JWM72RNI6IYQPLQEBDLZAJHKNNA, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:52.459781544 (741h17m52.330864579s) [noise=true , address=77.90.31.16:4755 , piece_id=FKWAU3TXE2NXHXFAG65K4JECFGN7U3BLAJ65WFQVKVZ2AUHSH3OA, seg_pos={0 0}, upload=648607]: Uploading piece... Jan 5 12:46:53.902949530 (741h17m53.774032575s) [upload=648607]: Next returned nil segment Jan 5 12:46:53.902966011 (741h17m53.774049066s) [upload=648607]: Waiting for error group managing segments... Jan 5 12:46:54.300576002 (741h17m54.171659047s) [noise=true , address=145.79.227.5:4239 , piece_id=FE4BDHDHQCCLJFGAIYS6SF47CQO353JRLHJVXBKR5K7YXWDSYCVA, seg_pos={0 0}, upload=648607]: Done uploading piece. err: <nil> Jan 5 12:46:54.303749955 (741h17m54.17483299s) [noise=true , address=145.79.227.40:4065 , piece_id=6O3F5YLQGESHONSZLQESSOVAIVIMWBOOSB7O6DSCDVWINNIK3DRA, seg_pos={0 0}, upload=648607]: Done uploading piece. err: <nil> Jan 5 12:46:54.330025167 (741h17m54.201108212s) [noise=true , address=145.79.227.35:4509 , piece_id=YYTKU2E7L2YDPXOFNM2WKCAZ6JWM72RNI6IYQPLQEBDLZAJHKNNA, seg_pos={0 0}, upload=648607]: Done uploading piece. err: <nil> Jan 5 12:46:54.384173935 (741h17m54.25525698s) [noise=true , address=145.79.227.5:4225 , piece_id=DVGNEBVWZU2C3TELA7QTR47VD772WER7C5GCXNXBGIEDWIT3SGXA, seg_pos={0 0}, upload=648607]: Done uploading piece. err: <nil> Jan 5 12:46:55.191882276 (741h17m55.062965321s) [noise=true , address=193.149.164.143:4696 , piece_id=7CTMW4C6NXQFOL45UVLL6B27JZC6PXUUKBCIIGRDUH6BROHSMDGA, seg_pos={0 0}, upload=648607]: Done uploading piece. err: <nil> Jan 5 12:46:55.208818095 (741h17m55.07990113s) [noise=true , address=145.79.227.44:4226 , piece_id=KYYYQE2OMDUBDE6E7NC66AXCQZRG5XRETSRFC5EBXR5YJBOT3OUQ, seg_pos={0 0}, upload=648607]: Done uploading piece. err: <nil> Jan 5 12:46:55.283782358 (741h17m55.154865393s) [noise=true , address=145.79.227.30:4793 , piece_id=XQZNHYAYWEKN2K464RPFSE5ERXDEZUWXBRYBAVDLRWYPBKRBLSAQ, seg_pos={0 0}, upload=648607]: Done uploading piece. err: <nil> Jan 5 12:46:55.336581105 (741h17m55.20766414s) [noise=true , address=145.79.227.33:4902 , piece_id=KSRXU2RUEKZ3GNS4WS5DZWDH7YXY2LILAE6ZH2TUZHQY5NRYZHSA, seg_pos={0 0}, upload=648607]: Done uploading piece. err: <nil> Jan 5 12:46:55.338557008 (741h17m55.209640043s) [noise=true , address=145.79.227.43:4182 , piece_id=2NTCCJ4CTQXAZB7FEVMFLIZG3IBWBFB5AHOVIJVWZBZADY6NIK6Q, seg_pos={0 0}, upload=648607]: Done uploading piece. err: <nil> Jan 5 12:46:55.367909029 (741h17m55.238992074s) [noise=true , address=145.79.227.32:4867 , piece_id=YYMB4ONF4FKGKRUKJ57W6FC57JOPKAI6QNPYBXGWKGF2YPYHL63A, seg_pos={0 0}, upload=648607]: Done uploading piece. err: <nil> Jan 5 12:46:55.763194593 (741h17m55.634277658s) [noise=true , address=145.79.227.29:4971 , piece_id=WOJZQ66XSFE4XRL2F3P5DAFN5BJP4JUYCLP6JDVXCMC4RRF77EHA, seg_pos={0 0}, upload=648607]: Done uploading piece. err: <nil> Jan 5 12:46:55.988827092 (741h17m55.859910127s) [noise=true , address=77.90.31.16:4755 , piece_id=FKWAU3TXE2NXHXFAG65K4JECFGN7U3BLAJ65WFQVKVZ2AUHSH3OA, seg_pos={0 0}, upload=648607]: Done uploading piece. err: <nil> Jan 5 12:46:56.004500646 (741h17m55.875583681s) [noise=true , address=77.90.31.19:4846 , piece_id=2HXBK56ZHRTWPRUDJ22SIBJJX4KKTWLJLQ4GRJCI7S7RPC25SYRA, seg_pos={0 0}, upload=648607]: Done uploading piece. err: <nil> Jan 5 12:46:56.155958997 (741h17m56.027042032s) [noise=true , address=77.90.31.10:4538 , piece_id=N23Q4IRVBYGEO6QJMUPJPKAB5MYWRB2ZVTPGTZZANUHHMWTOQP2A, seg_pos={0 0}, upload=648607]: Done uploading piece. err: <nil> Jan 5 12:46:56.155979025 (741h17m56.02706206s) [seg_pos={0 0}, upload=648607]: Segment reached optimal threshold of 14 pieces. Jan 5 12:46:56.156166783 (741h17m56.027249818s) [noise=true , address=77.90.31.13:4640 , piece_id=IQECUHYAXANWMWAR7S6FWVF63YJ5THLCYU7MK6U7KTAOQAXBDL7Q, seg_pos={0 0}, upload=648607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XjJpLBEZm5daHcBUmUyC7QpLVhu9Vm9ex9RYcYJWEuLGk527F): protocol: expected piece hash; context canceled; EOF Jan 5 12:46:56.156172383 (741h17m56.027255438s) [noise=true , address=77.90.31.13:4640 , piece_id=IQECUHYAXANWMWAR7S6FWVF63YJ5THLCYU7MK6U7KTAOQAXBDL7Q, seg_pos={0 0}, upload=648607]: Long tail cancellation due to optimal threshold Jan 5 12:46:56.156218020 (741h17m56.027301055s) [noise=true , address=45.91.116.11:4149 , piece_id=USMZ7R7R2D7AKO2NBNL4ANKTMD7YCFRCGMAJC4I3Q57WTVTP6TOA, seg_pos={0 0}, upload=648607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA): protocol: expected piece hash; context canceled; EOF Jan 5 12:46:56.156226887 (741h17m56.027309922s) [noise=true , address=45.91.116.11:4149 , piece_id=USMZ7R7R2D7AKO2NBNL4ANKTMD7YCFRCGMAJC4I3Q57WTVTP6TOA, seg_pos={0 0}, upload=648607]: Long tail cancellation due to optimal threshold Jan 5 12:46:56.156279557 (741h17m56.027362592s) [noise=true , address=64.49.13.169:4708 , piece_id=2B5EKXGQZU4B2AUV63P6IRTT3EN3EERCZTSII5HYIYLCWL5YE3AQ, seg_pos={0 0}, upload=648607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:48864->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:48864->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48864->64.49.13.169:4708: use of closed network connection Jan 5 12:46:56.156284537 (741h17m56.027367572s) [noise=true , address=64.49.13.169:4708 , piece_id=2B5EKXGQZU4B2AUV63P6IRTT3EN3EERCZTSII5HYIYLCWL5YE3AQ, seg_pos={0 0}, upload=648607]: Long tail cancellation due to optimal threshold Jan 5 12:46:56.156368115 (741h17m56.02745115s) [noise=true , address=45.91.116.8:4052 , piece_id=PHL2GIZ4KSJKRC3ZIBRSMFZ7AGCRPJ5YS4P3IUDJQ6IO35CZFK6Q, seg_pos={0 0}, upload=648607]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HHEiq4G4TCaHFJAN3DhL8SwXAFqcwRgYteSHmr1gNe2ThcmnS): protocol: write tcp 188.166.225.170:47976->45.91.116.8:4052: use of closed network connection; write tcp 188.166.225.170:47976->45.91.116.8:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47976->45.91.116.8:4052: use of closed network connection Jan 5 12:46:56.156374137 (741h17m56.027457172s) [noise=true , address=45.91.116.8:4052 , piece_id=PHL2GIZ4KSJKRC3ZIBRSMFZ7AGCRPJ5YS4P3IUDJQ6IO35CZFK6Q, seg_pos={0 0}, upload=648607]: Long tail cancellation due to optimal threshold Jan 5 12:46:56.156388585 (741h17m56.02747162s) [seg_pos={0 0}, upload=648607]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 5 12:46:56.156451345 (741h17m56.02753438s) [upload=648607]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:46:56.156455733 (741h17m56.027538778s) [upload=648607]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:46:56.448955928 (741h17m56.320038973s) [upload=648607]: Done uploading segments... Jan 5 12:46:56.448980375 (741h17m56.32006341s) [upload=648607]: Upload finished. err: <nil> Duration: 22.95093518s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [X.XXXX.XXXXXXXXXXXXXXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000521591s trace: https://jaeger.dev.storj.io/trace/5910c2967ebe0ed2 FAIL (2026-01-05 12:47:42 UTC) orbiter warning: context deadline exceeded Duration: 5.000542612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X..X...X..X.X..........X...XX...............XX.........X..] duration: 5.672637266s trace: https://jaeger.dev.storj.io/trace/28cbbf043d214b8f FAIL (2026-01-05 12:47:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5 12:45:21.016391497 (741h16m20.887474532s) [upload=648589]: Starting upload Jan 5 12:45:21.016449508 (741h16m20.887532543s) [upload=648589]: Done starting upload Jan 5 12:45:21.016606537 (741h16m20.887689572s) [upload=648589]: Uploading segments... Jan 5 12:45:21.016706537 (741h16m20.887789582s) [upload=648589]: Got next segment. Inline: false Jan 5 12:45:21.016739780 (741h16m20.887822825s) [upload=648589]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 5 12:45:21.016744811 (741h16m20.887827846s) [upload=648589]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 5 12:45:21.268595884 (741h16m21.139678929s) [seg_pos={0 0}, upload=648589]: Begin upload segment... Jan 5 12:45:21.268642443 (741h16m21.139725488s) [seg_pos={0 0}, upload=648589]: Done begin upload segment. Jan 5 12:45:21.268730540 (741h16m21.139813575s) [noise=true , address=145.79.227.22:4509 , piece_id=25JPBK3T7REYJP2WLO5K4LBLVIONOTRD5752G7MZWUHNXQ36562Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.270879172 (741h16m21.141962207s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.271442805 (741h16m21.14252584s) [noise=true , address=145.79.227.20:4437 , piece_id=VN5TKKWDYLGS7DBM5JKB6K2USRYFC7FUB4ZFHV3XJCJXZZ756G3Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.271905155 (741h16m21.14298819s) [noise=true , address=145.79.227.26:4653 , piece_id=DCMR6IJGZQELRMNI7LTMUPTCLSSL3U4JFR5TNZG2P36YSRN565TA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.272351776 (741h16m21.143434821s) [noise=true , address=145.79.227.48:4303 , piece_id=2D5IDHWWJXXZZH5WEL5UUDPWCCQJSYLB3HRWUHJAFHIJ6HQC7VZA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.272842941 (741h16m21.143925986s) [noise=true , address=145.79.227.28:4725 , piece_id=RHBNZJYMSWERVLZNUAEYSJNYCBWBKQZL72DX7AXGKU7V5PUAPM3Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.273284582 (741h16m21.144367627s) [noise=true , address=145.79.227.38:4979 , piece_id=BI5R7DQQFUDGR4AKJCXO6VUBKYKAZI5EAF5BWICVCDGD64NPQJQA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.273686899 (741h16m21.144769934s) [noise=true , address=45.91.116.14:4192 , piece_id=EUQFZAAJX5MN7TCT6U4K4VNL2UND56WESFXBDEL4K3FYBKZBNCNQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.274174016 (741h16m21.145257061s) [noise=true , address=145.79.227.50:4374 , piece_id=NFZSOJMZOLZKF7V52F37WAPSN6V7ZFYTPWY43ICQWIKDPPP4MHUA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.274699497 (741h16m21.145782532s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.275224597 (741h16m21.146307632s) [noise=true , address=145.79.227.5:4222 , piece_id=B2LPQWSTK6ESRTVV5LGUJZVXL277TV3FTCN6FFGXGVUGCRPBVYUA, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.275731632 (741h16m21.146814677s) [noise=true , address=145.79.227.42:4121 , piece_id=RZNCW5NERG453IQLF564CPOUEEPLFK4GIZUWCV4WHHVZ5DIW66GQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.277290099 (741h16m21.148373134s) [noise=true , address=77.90.31.19:4842 , piece_id=F7E2LUJMCYFWHOGNSM2VXUTTMCFCP37VZQLCLP5VD2FTN5CQY7NQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.279405628 (741h16m21.150488673s) [noise=true , address=77.90.31.18:4808 , piece_id=VLTWI3TUQ5DZWJS7KYATLVDHP6RK3WPRY4I65KVAHJUPHBMOCY7Q, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.281446874 (741h16m21.152529919s) [noise=true , address=145.79.227.37:4565 , piece_id=VCJGFGJDQSLNSKMOAKDZZ5QFXVPA3DEWRCWFWRT6EIKRHZJZGR5A, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:21.283204051 (741h16m21.154287096s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Uploading piece... Jan 5 12:45:22.966583542 (741h16m22.837666587s) [upload=648589]: Next returned nil segment Jan 5 12:45:22.966601036 (741h16m22.837684081s) [upload=648589]: Waiting for error group managing segments... Jan 5 12:45:23.259998799 (741h16m23.131081834s) [noise=true , address=145.79.227.22:4509 , piece_id=25JPBK3T7REYJP2WLO5K4LBLVIONOTRD5752G7MZWUHNXQ36562Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.266672608 (741h16m23.137755653s) [noise=true , address=145.79.227.28:4725 , piece_id=RHBNZJYMSWERVLZNUAEYSJNYCBWBKQZL72DX7AXGKU7V5PUAPM3Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.268418462 (741h16m23.139501507s) [noise=true , address=145.79.227.5:4222 , piece_id=B2LPQWSTK6ESRTVV5LGUJZVXL277TV3FTCN6FFGXGVUGCRPBVYUA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.275051953 (741h16m23.146134998s) [noise=true , address=145.79.227.38:4979 , piece_id=BI5R7DQQFUDGR4AKJCXO6VUBKYKAZI5EAF5BWICVCDGD64NPQJQA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.277622617 (741h16m23.148705662s) [noise=true , address=145.79.227.37:4565 , piece_id=VCJGFGJDQSLNSKMOAKDZZ5QFXVPA3DEWRCWFWRT6EIKRHZJZGR5A, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.288675784 (741h16m23.159758819s) [noise=true , address=145.79.227.20:4437 , piece_id=VN5TKKWDYLGS7DBM5JKB6K2USRYFC7FUB4ZFHV3XJCJXZZ756G3Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.288731430 (741h16m23.159814475s) [noise=true , address=145.79.227.48:4303 , piece_id=2D5IDHWWJXXZZH5WEL5UUDPWCCQJSYLB3HRWUHJAFHIJ6HQC7VZA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.290197542 (741h16m23.161280587s) [noise=true , address=145.79.227.26:4653 , piece_id=DCMR6IJGZQELRMNI7LTMUPTCLSSL3U4JFR5TNZG2P36YSRN565TA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.291884975 (741h16m23.16296803s) [noise=true , address=145.79.227.50:4374 , piece_id=NFZSOJMZOLZKF7V52F37WAPSN6V7ZFYTPWY43ICQWIKDPPP4MHUA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.292911229 (741h16m23.163994274s) [noise=true , address=145.79.227.42:4121 , piece_id=RZNCW5NERG453IQLF564CPOUEEPLFK4GIZUWCV4WHHVZ5DIW66GQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:23.808597866 (741h16m23.679680911s) [noise=true , address=45.91.116.14:4192 , piece_id=EUQFZAAJX5MN7TCT6U4K4VNL2UND56WESFXBDEL4K3FYBKZBNCNQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:24.376048792 (741h16m24.247131837s) [noise=true , address=77.90.31.18:4808 , piece_id=VLTWI3TUQ5DZWJS7KYATLVDHP6RK3WPRY4I65KVAHJUPHBMOCY7Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:25.047325497 (741h16m24.918408542s) [noise=true , address=77.90.31.19:4842 , piece_id=F7E2LUJMCYFWHOGNSM2VXUTTMCFCP37VZQLCLP5VD2FTN5CQY7NQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: <nil> Jan 5 12:45:25.047352187 (741h16m24.918435222s) [seg_pos={0 0}, upload=648589]: Segment reached optimal threshold of 13 pieces. Jan 5 12:45:25.047587886 (741h16m24.918670931s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Jan 5 12:45:25.047597243 (741h16m24.918680288s) [noise=true , address=45.91.116.7:4009 , piece_id=C4N7Q5U5HS6FGVOFQQZXMBDSOUPWI2CYW3APSW6HCIHOVBX7FK7Q, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047601892 (741h16m24.918684947s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48224->45.91.116.6:4974: use of closed network connection Jan 5 12:45:25.047610149 (741h16m24.918693194s) [noise=true , address=45.91.116.6:4974 , piece_id=2GV3PI2DNTPAS3GKGTL6P6HUCOSAQ3FD2PUHY7DTLCK23QZD7FKA, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047827893 (741h16m24.918910938s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Jan 5 12:45:25.047837501 (741h16m24.918920546s) [noise=true , address=77.90.31.21:4880 , piece_id=M4GAKWY2R6HLZSRD6M2JE3ZL72KK67V5GF64R7PHEA3FBW6F6BFQ, seg_pos={0 0}, upload=648589]: Long tail cancellation due to optimal threshold Jan 5 12:45:25.047856027 (741h16m24.918939072s) [seg_pos={0 0}, upload=648589]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 5 12:45:25.047959614 (741h16m24.919042659s) [upload=648589]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 5 12:45:25.047967589 (741h16m24.919050624s) [upload=648589]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 5 12:45:25.345497861 (741h16m25.216580906s) [upload=648589]: Done uploading segments... Jan 5 12:45:25.345532617 (741h16m25.216615662s) [upload=648589]: Upload finished. err: <nil> Duration: 5.672663426s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.............X....................] duration: 22.362519939s trace: https://jaeger.dev.storj.io/trace/5f0936145b5399cd FAIL (2026-01-05 12:47:23 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 22.362548174s edge gateway us-select-1

1
...
280
281
282
283
284
...
8097