Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 879930 reports (Page 11491 of 17599)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.XX] duration: 5.00088629s trace: https://jaeger.dev.storj.io/trace/603bc903fd267b16 FAIL (2025-12-16 14:19:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000908803s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................X.XX....X....................X..X.] duration: 13.80047178s trace: https://jaeger.dev.storj.io/trace/7e8b1ea0d3c2b5e FAIL (2025-12-16 14:19:40 UTC) orbiter warning: context deadline exceeded Duration: 13.800493642s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................X.XX....X....................X..X] duration: 30.000962116s trace: https://jaeger.dev.storj.io/trace/7e8b1ea0d3c2b5e FAIL (2025-12-16 14:19:26 UTC) orbiter warning: context deadline exceeded Duration: 30.000984368s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [X.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.X] duration: 5.0008689s trace: https://jaeger.dev.storj.io/trace/65471482ca9f4666 FAIL (2025-12-16 14:19:23 UTC) orbiter warning: context deadline exceeded Duration: 5.000894078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 42.710675909s trace: https://jaeger.dev.storj.io/trace/ba2c44dde12044 FAIL (2025-12-16 14:19:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=229725]: Uploading piece... Dec 16 13:30:17.600801266 (262h1m17.471884321s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.600906235 (262h1m17.471989281s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601269747 (262h1m17.472352782s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601276790 (262h1m17.472359835s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601630013 (262h1m17.472713048s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601640714 (262h1m17.472723749s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601965091 (262h1m17.473048126s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602047147 (262h1m17.473130192s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602246808 (262h1m17.473329853s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602601232 (262h1m17.473684277s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602716132 (262h1m17.473799167s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602996997 (262h1m17.474080042s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603027615 (262h1m17.47411065s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603255378 (262h1m17.474338413s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603501387 (262h1m17.474584432s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603959158 (262h1m17.475042203s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.604485590 (262h1m17.475568635s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:19.111089229 (262h1m18.982172264s) [upload=229725]: Next returned nil segment Dec 16 13:30:19.111104708 (262h1m18.982187753s) [upload=229725]: Waiting for error group managing segments... Dec 16 13:30:19.527483020 (262h1m19.398566075s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.621753699 (262h1m19.492836744s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.670564204 (262h1m19.541647249s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.672302895 (262h1m19.54338594s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.716239789 (262h1m19.587322834s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.735358003 (262h1m19.606441048s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.750731367 (262h1m19.621814412s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.764604575 (262h1m19.63568761s) [noise=true , address=45.91.116.7:4029 , piece_id=NZ5HRBRJ3QMNZMXJLKD5NPC3ZZFAS7VIDQNXPURRPWOCMPTXFUNQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.796960486 (262h1m19.668043531s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.812007538 (262h1m19.683090573s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.032271213 (262h1m19.903354248s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.099742841 (262h1m19.970825886s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.231931353 (262h1m20.103014398s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394678143 (262h1m20.265761188s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394706487 (262h1m20.265789532s) [seg_pos={0 0}, upload=229725]: Segment reached optimal threshold of 14 pieces. Dec 16 13:30:20.394971411 (262h1m20.266054466s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.394981671 (262h1m20.266064716s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395367956 (262h1m20.266451001s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395378747 (262h1m20.266461792s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395542348 (262h1m20.266625393s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395565903 (262h1m20.266648948s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395688968 (262h1m20.266772013s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395695741 (262h1m20.266778786s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395718144 (262h1m20.266801189s) [seg_pos={0 0}, upload=229725]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 13:30:20.395862569 (262h1m20.266945604s) [upload=229725]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:30:20.395896854 (262h1m20.266979899s) [upload=229725]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:30:20.679088774 (262h1m20.550171819s) [upload=229725]: Done uploading segments... Dec 16 13:30:20.679109594 (262h1m20.550192629s) [upload=229725]: Upload finished. err: <nil> Duration: 42.710703893s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................X.XX....X....................X..] duration: 18.579257977s trace: https://jaeger.dev.storj.io/trace/7fa93c16b817c086 FAIL (2025-12-16 14:18:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.57928064s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................X....X......X......X....] duration: 6.089819171s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:18:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 6.089847033s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................X.XX....X....................X.] duration: 16.108221264s trace: https://jaeger.dev.storj.io/trace/7fa93c16b817c086 FAIL (2025-12-16 14:18:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.108250361s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X....X......X......X...] duration: 5.750543664s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:18:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 5.750574324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 41.276211428s trace: https://jaeger.dev.storj.io/trace/ba2c44dde12044 FAIL (2025-12-16 14:18:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=229725]: Uploading piece... Dec 16 13:30:17.600801266 (262h1m17.471884321s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.600906235 (262h1m17.471989281s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601269747 (262h1m17.472352782s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601276790 (262h1m17.472359835s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601630013 (262h1m17.472713048s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601640714 (262h1m17.472723749s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601965091 (262h1m17.473048126s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602047147 (262h1m17.473130192s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602246808 (262h1m17.473329853s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602601232 (262h1m17.473684277s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602716132 (262h1m17.473799167s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602996997 (262h1m17.474080042s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603027615 (262h1m17.47411065s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603255378 (262h1m17.474338413s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603501387 (262h1m17.474584432s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603959158 (262h1m17.475042203s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.604485590 (262h1m17.475568635s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:19.111089229 (262h1m18.982172264s) [upload=229725]: Next returned nil segment Dec 16 13:30:19.111104708 (262h1m18.982187753s) [upload=229725]: Waiting for error group managing segments... Dec 16 13:30:19.527483020 (262h1m19.398566075s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.621753699 (262h1m19.492836744s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.670564204 (262h1m19.541647249s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.672302895 (262h1m19.54338594s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.716239789 (262h1m19.587322834s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.735358003 (262h1m19.606441048s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.750731367 (262h1m19.621814412s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.764604575 (262h1m19.63568761s) [noise=true , address=45.91.116.7:4029 , piece_id=NZ5HRBRJ3QMNZMXJLKD5NPC3ZZFAS7VIDQNXPURRPWOCMPTXFUNQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.796960486 (262h1m19.668043531s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.812007538 (262h1m19.683090573s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.032271213 (262h1m19.903354248s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.099742841 (262h1m19.970825886s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.231931353 (262h1m20.103014398s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394678143 (262h1m20.265761188s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394706487 (262h1m20.265789532s) [seg_pos={0 0}, upload=229725]: Segment reached optimal threshold of 14 pieces. Dec 16 13:30:20.394971411 (262h1m20.266054466s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.394981671 (262h1m20.266064716s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395367956 (262h1m20.266451001s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395378747 (262h1m20.266461792s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395542348 (262h1m20.266625393s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395565903 (262h1m20.266648948s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395688968 (262h1m20.266772013s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395695741 (262h1m20.266778786s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395718144 (262h1m20.266801189s) [seg_pos={0 0}, upload=229725]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 13:30:20.395862569 (262h1m20.266945604s) [upload=229725]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:30:20.395896854 (262h1m20.266979899s) [upload=229725]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:30:20.679088774 (262h1m20.550171819s) [upload=229725]: Done uploading segments... Dec 16 13:30:20.679109594 (262h1m20.550192629s) [upload=229725]: Upload finished. err: <nil> Duration: 41.276239721s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XX.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX.] duration: 3.733002627s trace: https://jaeger.dev.storj.io/trace/2310ce4ca9162552 FAIL (2025-12-16 14:18:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.733254035s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................X.XX....X....................X] duration: 30.001520684s trace: https://jaeger.dev.storj.io/trace/7fa93c16b817c086 FAIL (2025-12-16 14:17:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001543907s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXX.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..XX] duration: 5.001127265s trace: https://jaeger.dev.storj.io/trace/2310ce4ca9162552 FAIL (2025-12-16 14:17:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001150187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................X....X......X......X..] duration: 5.76199051s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:17:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 5.762018704s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 40.256571308s trace: https://jaeger.dev.storj.io/trace/ba2c44dde12044 FAIL (2025-12-16 14:17:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=229725]: Uploading piece... Dec 16 13:30:17.600801266 (262h1m17.471884321s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.600906235 (262h1m17.471989281s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601269747 (262h1m17.472352782s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601276790 (262h1m17.472359835s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601630013 (262h1m17.472713048s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601640714 (262h1m17.472723749s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601965091 (262h1m17.473048126s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602047147 (262h1m17.473130192s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602246808 (262h1m17.473329853s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602601232 (262h1m17.473684277s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602716132 (262h1m17.473799167s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602996997 (262h1m17.474080042s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603027615 (262h1m17.47411065s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603255378 (262h1m17.474338413s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603501387 (262h1m17.474584432s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603959158 (262h1m17.475042203s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.604485590 (262h1m17.475568635s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:19.111089229 (262h1m18.982172264s) [upload=229725]: Next returned nil segment Dec 16 13:30:19.111104708 (262h1m18.982187753s) [upload=229725]: Waiting for error group managing segments... Dec 16 13:30:19.527483020 (262h1m19.398566075s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.621753699 (262h1m19.492836744s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.670564204 (262h1m19.541647249s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.672302895 (262h1m19.54338594s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.716239789 (262h1m19.587322834s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.735358003 (262h1m19.606441048s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.750731367 (262h1m19.621814412s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.764604575 (262h1m19.63568761s) [noise=true , address=45.91.116.7:4029 , piece_id=NZ5HRBRJ3QMNZMXJLKD5NPC3ZZFAS7VIDQNXPURRPWOCMPTXFUNQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.796960486 (262h1m19.668043531s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.812007538 (262h1m19.683090573s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.032271213 (262h1m19.903354248s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.099742841 (262h1m19.970825886s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.231931353 (262h1m20.103014398s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394678143 (262h1m20.265761188s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394706487 (262h1m20.265789532s) [seg_pos={0 0}, upload=229725]: Segment reached optimal threshold of 14 pieces. Dec 16 13:30:20.394971411 (262h1m20.266054466s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.394981671 (262h1m20.266064716s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395367956 (262h1m20.266451001s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395378747 (262h1m20.266461792s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395542348 (262h1m20.266625393s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395565903 (262h1m20.266648948s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395688968 (262h1m20.266772013s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395695741 (262h1m20.266778786s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395718144 (262h1m20.266801189s) [seg_pos={0 0}, upload=229725]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 13:30:20.395862569 (262h1m20.266945604s) [upload=229725]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:30:20.395896854 (262h1m20.266979899s) [upload=229725]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:30:20.679088774 (262h1m20.550171819s) [upload=229725]: Done uploading segments... Dec 16 13:30:20.679109594 (262h1m20.550192629s) [upload=229725]: Upload finished. err: <nil> Duration: 40.256589394s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [.XXX.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..X] duration: 5.001715433s trace: https://jaeger.dev.storj.io/trace/109e6a61eada83a2 FAIL (2025-12-16 14:17:10 UTC) orbiter warning: context deadline exceeded Duration: 5.001739649s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................X....X......X......X.] duration: 5.929003326s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:16:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 5.92903168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 42.943209355s trace: https://jaeger.dev.storj.io/trace/ba2c44dde12044 FAIL (2025-12-16 14:16:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=229725]: Uploading piece... Dec 16 13:30:17.600801266 (262h1m17.471884321s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.600906235 (262h1m17.471989281s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601269747 (262h1m17.472352782s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601276790 (262h1m17.472359835s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601630013 (262h1m17.472713048s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601640714 (262h1m17.472723749s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601965091 (262h1m17.473048126s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602047147 (262h1m17.473130192s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602246808 (262h1m17.473329853s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602601232 (262h1m17.473684277s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602716132 (262h1m17.473799167s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602996997 (262h1m17.474080042s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603027615 (262h1m17.47411065s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603255378 (262h1m17.474338413s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603501387 (262h1m17.474584432s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603959158 (262h1m17.475042203s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.604485590 (262h1m17.475568635s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:19.111089229 (262h1m18.982172264s) [upload=229725]: Next returned nil segment Dec 16 13:30:19.111104708 (262h1m18.982187753s) [upload=229725]: Waiting for error group managing segments... Dec 16 13:30:19.527483020 (262h1m19.398566075s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.621753699 (262h1m19.492836744s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.670564204 (262h1m19.541647249s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.672302895 (262h1m19.54338594s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.716239789 (262h1m19.587322834s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.735358003 (262h1m19.606441048s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.750731367 (262h1m19.621814412s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.764604575 (262h1m19.63568761s) [noise=true , address=45.91.116.7:4029 , piece_id=NZ5HRBRJ3QMNZMXJLKD5NPC3ZZFAS7VIDQNXPURRPWOCMPTXFUNQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.796960486 (262h1m19.668043531s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.812007538 (262h1m19.683090573s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.032271213 (262h1m19.903354248s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.099742841 (262h1m19.970825886s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.231931353 (262h1m20.103014398s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394678143 (262h1m20.265761188s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394706487 (262h1m20.265789532s) [seg_pos={0 0}, upload=229725]: Segment reached optimal threshold of 14 pieces. Dec 16 13:30:20.394971411 (262h1m20.266054466s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.394981671 (262h1m20.266064716s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395367956 (262h1m20.266451001s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395378747 (262h1m20.266461792s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395542348 (262h1m20.266625393s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395565903 (262h1m20.266648948s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395688968 (262h1m20.266772013s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395695741 (262h1m20.266778786s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395718144 (262h1m20.266801189s) [seg_pos={0 0}, upload=229725]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 13:30:20.395862569 (262h1m20.266945604s) [upload=229725]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:30:20.395896854 (262h1m20.266979899s) [upload=229725]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:30:20.679088774 (262h1m20.550171819s) [upload=229725]: Done uploading segments... Dec 16 13:30:20.679109594 (262h1m20.550192629s) [upload=229725]: Upload finished. err: <nil> Duration: 42.94323786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................X.XX....X....................] duration: 10.921906209s trace: https://jaeger.dev.storj.io/trace/24f03ce3e3f5cb2d FAIL (2025-12-16 14:16:26 UTC) orbiter warning: context deadline exceeded Duration: 10.922023402s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [X.XXX.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX..] duration: 4.301005029s trace: https://jaeger.dev.storj.io/trace/6b81613ec6bde182 FAIL (2025-12-16 14:16:26 UTC) orbiter warning: context deadline exceeded Duration: 4.301031719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................X....X......X......X] duration: 30.003655455s trace: https://jaeger.dev.storj.io/trace/2e51804a22ca1f31 FAIL (2025-12-16 14:16:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 14:15:38.916700212 (262h46m38.787783258s) [upload=230361]: Starting upload Dec 16 14:15:38.916764094 (262h46m38.787847139s) [upload=230361]: Done starting upload Dec 16 14:15:38.916983392 (262h46m38.788066427s) [upload=230361]: Uploading segments... Dec 16 14:15:38.917032755 (262h46m38.7881158s) [upload=230361]: Got next segment. Inline: false Dec 16 14:15:38.917062462 (262h46m38.788145497s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:15:38.917067151 (262h46m38.788150186s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:15:39.184763536 (262h46m39.055846571s) [seg_pos={0 0}, upload=230361]: Begin upload segment... Dec 16 14:15:39.184819182 (262h46m39.055902217s) [seg_pos={0 0}, upload=230361]: Done begin upload segment. Dec 16 14:15:39.184852786 (262h46m39.055935831s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.184891560 (262h46m39.055974605s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185347768 (262h46m39.056430803s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185356425 (262h46m39.05643947s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185720709 (262h46m39.056803754s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.185742420 (262h46m39.056825466s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186016762 (262h46m39.057099797s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186119838 (262h46m39.057202873s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186291095 (262h46m39.05737413s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186593571 (262h46m39.057676616s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186680997 (262h46m39.057764032s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186947575 (262h46m39.05803061s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.186960228 (262h46m39.058043264s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187220485 (262h46m39.05830352s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187285258 (262h46m39.058368303s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:39.187621067 (262h46m39.058704112s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Uploading piece... Dec 16 14:15:40.333794157 (262h46m40.204877202s) [noise=true , address=77.90.31.10:4517 , piece_id=TTAVQGV2GQMQVCCGLK5XA7YM2NNFCH7NMSRU6HOPF7SXTEHJW4XA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 16 14:15:40.890201963 (262h46m40.761285028s) [upload=230361]: Next returned nil segment Dec 16 14:15:40.890219006 (262h46m40.761302051s) [upload=230361]: Waiting for error group managing segments... Dec 16 14:15:41.175938840 (262h46m41.047021885s) [noise=true , address=145.79.227.16:4910 , piece_id=ZJ63P5D3IG4RMMBXKVZMVLAXTBQOMGTAPSVRQO4RYNWAJA5SAFBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.176302473 (262h46m41.047385518s) [noise=true , address=145.79.227.39:4015 , piece_id=KN5RSQLD5DKRQDSXW3CLKKOC5R7LDMWTO5PH2I5R57UYLIPUXOMA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.178137478 (262h46m41.049220523s) [noise=true , address=145.79.227.43:4157 , piece_id=V2XDIFJWYWKXFT6WISW7WYF42PX7L7HIFTO3KJFRAYHK4JTMKEAA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.199331023 (262h46m41.070414058s) [noise=true , address=145.79.227.32:4834 , piece_id=VHPPL3BK4MIZAEDCWH42YIHKYZC46DB3IHS5ZWLH2ALCRXTLOIZA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200184949 (262h46m41.071267984s) [noise=true , address=145.79.227.31:4799 , piece_id=L7YRENYHSIMKPMUGB355RWLBNBUFB6TQ6O5LMVIWYPLRNNXXNYWA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.200629837 (262h46m41.071712872s) [noise=true , address=145.79.227.37:4565 , piece_id=BIGQTMKHW5L575C6M4PMC3M5QWDFFZ7GTCGQJTDLB4UDGJTVQLAQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.203198097 (262h46m41.074281142s) [noise=true , address=145.79.227.18:4007 , piece_id=4ND7XUSOXSH4LPPWPQAJKM5QNDAQ2XLUVLF2WDMQGZCM3YHMXW7Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.209402041 (262h46m41.080485086s) [noise=true , address=145.79.227.41:4085 , piece_id=JNOTJDQTB2T3KKV2247VBU6FTR4BT756V3WXWS322RXUEOZU3HSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.307731509 (262h46m41.178814554s) [noise=true , address=145.79.227.46:4603 , piece_id=AJCXST2I7WWNHWSQTXXFZB6APIDMXJHWOOR743UGCLQ3NP6QUJTQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.311024429 (262h46m41.182107474s) [noise=true , address=145.79.227.27:4691 , piece_id=MVF5ODWIMNMMOYZ5MVHNRFPVD4TTUXDYARTL7RG6XEA6X2CYABPA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.328720826 (262h46m41.199803872s) [noise=true , address=145.79.227.11:4729 , piece_id=EKSQM3YA6Z32BJ5QTGXTCIBVEZ4UTR7KTBXJAA253O25SKRPGD4Q, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.432525339 (262h46m41.303608384s) [noise=true , address=77.90.31.11:4556 , piece_id=ST7P7XH3ZVO4JXJ7VXJQ4JVN5NQSMJLBXAYIYTJQBYVH2HP4GCBQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515147718 (262h46m41.386230763s) [noise=true , address=45.91.116.11:4120 , piece_id=WY3XLMX3QIOWLJO5MSFR7RTW5JX77F7KVQFKD6NK2VLF4255PJUQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: <nil> Dec 16 14:15:41.515171093 (262h46m41.386254138s) [seg_pos={0 0}, upload=230361]: Segment reached optimal threshold of 13 pieces. Dec 16 14:15:41.515425878 (262h46m41.386508933s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: expected piece hash; context canceled; EOF Dec 16 14:15:41.515437060 (262h46m41.386520105s) [noise=true , address=77.90.31.16:4734 , piece_id=XNTDQKROMZMNU5D3IPTVX6YYZTKRCQEZKPPGXLA6P4ZR6ICG5LSQ, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515806893 (262h46m41.386889938s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47302->45.91.116.7:4012: use of closed network connection Dec 16 14:15:41.515816142 (262h46m41.386899197s) [noise=true , address=45.91.116.7:4012 , piece_id=TICNXMR4ZD5NIOZJBACFIZ7PQY7JNP2HRB4WMQHNEYSYGWXWSKUA, seg_pos={0 0}, upload=230361]: Long tail cancellation due to optimal threshold Dec 16 14:15:41.515838043 (262h46m41.386921108s) [seg_pos={0 0}, upload=230361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:15:41.515933535 (262h46m41.38701658s) [upload=230361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:15:41.515940738 (262h46m41.387023793s) [upload=230361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:15:41.848268965 (262h46m41.71935201s) [upload=230361]: Done uploading segments... Dec 16 14:15:41.848293422 (262h46m41.719376467s) [upload=230361]: Upload finished. err: <nil> Duration: 30.003678469s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................X.XX....X...................] duration: 9.091215827s trace: https://jaeger.dev.storj.io/trace/24f03ce3e3f5cb2d FAIL (2025-12-16 14:15:46 UTC) orbiter warning: context deadline exceeded Duration: 9.091241537s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................X.XX....X..................] duration: 25.698044056s trace: https://jaeger.dev.storj.io/trace/24f03ce3e3f5cb2d FAIL (2025-12-16 14:15:35 UTC) orbiter warning: context deadline exceeded Duration: 25.698072309s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 44.424520026s trace: https://jaeger.dev.storj.io/trace/ba2c44dde12044 FAIL (2025-12-16 14:15:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=229725]: Uploading piece... Dec 16 13:30:17.600801266 (262h1m17.471884321s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.600906235 (262h1m17.471989281s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601269747 (262h1m17.472352782s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601276790 (262h1m17.472359835s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601630013 (262h1m17.472713048s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601640714 (262h1m17.472723749s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601965091 (262h1m17.473048126s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602047147 (262h1m17.473130192s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602246808 (262h1m17.473329853s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602601232 (262h1m17.473684277s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602716132 (262h1m17.473799167s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602996997 (262h1m17.474080042s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603027615 (262h1m17.47411065s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603255378 (262h1m17.474338413s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603501387 (262h1m17.474584432s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603959158 (262h1m17.475042203s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.604485590 (262h1m17.475568635s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:19.111089229 (262h1m18.982172264s) [upload=229725]: Next returned nil segment Dec 16 13:30:19.111104708 (262h1m18.982187753s) [upload=229725]: Waiting for error group managing segments... Dec 16 13:30:19.527483020 (262h1m19.398566075s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.621753699 (262h1m19.492836744s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.670564204 (262h1m19.541647249s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.672302895 (262h1m19.54338594s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.716239789 (262h1m19.587322834s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.735358003 (262h1m19.606441048s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.750731367 (262h1m19.621814412s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.764604575 (262h1m19.63568761s) [noise=true , address=45.91.116.7:4029 , piece_id=NZ5HRBRJ3QMNZMXJLKD5NPC3ZZFAS7VIDQNXPURRPWOCMPTXFUNQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.796960486 (262h1m19.668043531s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.812007538 (262h1m19.683090573s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.032271213 (262h1m19.903354248s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.099742841 (262h1m19.970825886s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.231931353 (262h1m20.103014398s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394678143 (262h1m20.265761188s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394706487 (262h1m20.265789532s) [seg_pos={0 0}, upload=229725]: Segment reached optimal threshold of 14 pieces. Dec 16 13:30:20.394971411 (262h1m20.266054466s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.394981671 (262h1m20.266064716s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395367956 (262h1m20.266451001s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395378747 (262h1m20.266461792s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395542348 (262h1m20.266625393s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395565903 (262h1m20.266648948s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395688968 (262h1m20.266772013s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395695741 (262h1m20.266778786s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395718144 (262h1m20.266801189s) [seg_pos={0 0}, upload=229725]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 13:30:20.395862569 (262h1m20.266945604s) [upload=229725]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:30:20.395896854 (262h1m20.266979899s) [upload=229725]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:30:20.679088774 (262h1m20.550171819s) [upload=229725]: Done uploading segments... Dec 16 13:30:20.679109594 (262h1m20.550192629s) [upload=229725]: Upload finished. err: <nil> Duration: 44.424561455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [.X.XXX.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX.] duration: 3.813971586s trace: https://jaeger.dev.storj.io/trace/6b81613ec6bde182 FAIL (2025-12-16 14:15:15 UTC) orbiter warning: context deadline exceeded Duration: 3.814064623s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [X.X.XXX.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXXX] duration: 5.00173213s trace: https://jaeger.dev.storj.io/trace/6b81613ec6bde182 FAIL (2025-12-16 14:14:51 UTC) orbiter warning: context deadline exceeded Duration: 5.001757487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X....X......X......] duration: 5.908422417s trace: https://jaeger.dev.storj.io/trace/e49c6365710000b FAIL (2025-12-16 14:14:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ) context deadline exceeded ======LOGS====== Dec 16 14:10:32.985073926 (262h41m32.856156961s) [upload=230290]: Starting upload Dec 16 14:10:32.985154330 (262h41m32.856237375s) [upload=230290]: Done starting upload Dec 16 14:10:32.985233330 (262h41m32.856316365s) [upload=230290]: Uploading segments... Dec 16 14:10:32.985485239 (262h41m32.856568294s) [upload=230290]: Got next segment. Inline: false Dec 16 14:10:32.985601450 (262h41m32.856684485s) [upload=230290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:10:32.985608184 (262h41m32.856691229s) [upload=230290]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:10:33.255392794 (262h41m33.126475829s) [seg_pos={0 0}, upload=230290]: Begin upload segment... Dec 16 14:10:33.255499337 (262h41m33.126582372s) [seg_pos={0 0}, upload=230290]: Done begin upload segment. Dec 16 14:10:33.255557117 (262h41m33.126640152s) [noise=true , address=145.79.227.51:4410 , piece_id=56VWGQ2SOLXWSNIXJMRI2VQ7VXYILL5EZN45IPSK6ZCWEJA4U2RA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.255585682 (262h41m33.126668727s) [noise=true , address=145.79.227.23:4545 , piece_id=PNRKCEAPMKBQTQMP7OOTS3S5MEW74VFREWCHZ5XAC7IDWF6S5MWA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.256346882 (262h41m33.127429917s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.256562532 (262h41m33.127645577s) [noise=true , address=145.79.227.40:4050 , piece_id=QPPTYML43XJRZCVQ5QKVTJMWOH4ETB6P2QUYQTPERBJ73OROLTWA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.257542648 (262h41m33.128625693s) [noise=true , address=45.91.116.6:4976 , piece_id=GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.258212524 (262h41m33.12929556s) [noise=true , address=145.79.227.17:4972 , piece_id=SS2ZYEXA2OWPY4MXJIT53HXHT5BZ5FBNFNECV3JVIPRTZCWRLXHQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.258731993 (262h41m33.129815028s) [noise=true , address=145.79.227.9:4365 , piece_id=7CHLK56ZSQ3VMLCHTGWEPXTYSOI7L6IX5IK7UUTFPYK434Y2BJ5Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259079935 (262h41m33.13016297s) [noise=true , address=145.79.227.45:4231 , piece_id=GI24XZVNFRHGDWBWFE3KE7W5OHBVEHLJ3B2BMCP6PNKGWJRAAYMA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259188884 (262h41m33.130271919s) [noise=true , address=45.91.116.14:4192 , piece_id=WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259521847 (262h41m33.130604892s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259824013 (262h41m33.130907048s) [noise=true , address=145.79.227.38:4979 , piece_id=4NOR7PSPCHWIKHA5GGGPJA3FXNCNEXKWKOLBMORBEZG4NKXRTR4Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260168439 (262h41m33.131251484s) [noise=true , address=145.79.227.32:4836 , piece_id=F6L5YJ773REOLNMASD2JD7BTGFKB2UBXZMBPYERZ5I6DTYY74OMA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260470092 (262h41m33.131553137s) [noise=true , address=145.79.227.15:4876 , piece_id=DACMMDUBQG2TF2VERLEJ5CJXEWKGYTYTQ376KGICJNMCFFG4RWEA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260751117 (262h41m33.131834163s) [noise=true , address=145.79.227.50:4374 , piece_id=O2T3JYB2C6RRAB3NXMQHBXF4EEGQES2KYPBGR6JM6SIIGFYNWPZQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.261047001 (262h41m33.132130036s) [noise=true , address=145.79.227.47:4267 , piece_id=CRCIH27Q7AYOJED5Q5Y3LB4GXOWJNEVAMXOGBTHEF6WKBGVW2EHQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.261378823 (262h41m33.132461858s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:34.973936146 (262h41m34.845019181s) [upload=230290]: Next returned nil segment Dec 16 14:10:34.973950845 (262h41m34.84503388s) [upload=230290]: Waiting for error group managing segments... Dec 16 14:10:35.255716630 (262h41m35.126799675s) [noise=true , address=145.79.227.45:4231 , piece_id=GI24XZVNFRHGDWBWFE3KE7W5OHBVEHLJ3B2BMCP6PNKGWJRAAYMA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.260286111 (262h41m35.131369156s) [noise=true , address=145.79.227.47:4267 , piece_id=CRCIH27Q7AYOJED5Q5Y3LB4GXOWJNEVAMXOGBTHEF6WKBGVW2EHQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.266206785 (262h41m35.13728982s) [noise=true , address=145.79.227.17:4972 , piece_id=SS2ZYEXA2OWPY4MXJIT53HXHT5BZ5FBNFNECV3JVIPRTZCWRLXHQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.269766433 (262h41m35.140849468s) [noise=true , address=145.79.227.15:4876 , piece_id=DACMMDUBQG2TF2VERLEJ5CJXEWKGYTYTQ376KGICJNMCFFG4RWEA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.277099697 (262h41m35.148182742s) [noise=true , address=145.79.227.50:4374 , piece_id=O2T3JYB2C6RRAB3NXMQHBXF4EEGQES2KYPBGR6JM6SIIGFYNWPZQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.277901374 (262h41m35.148984409s) [noise=true , address=145.79.227.32:4836 , piece_id=F6L5YJ773REOLNMASD2JD7BTGFKB2UBXZMBPYERZ5I6DTYY74OMA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.335067849 (262h41m35.206150884s) [noise=true , address=145.79.227.9:4365 , piece_id=7CHLK56ZSQ3VMLCHTGWEPXTYSOI7L6IX5IK7UUTFPYK434Y2BJ5Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.355162373 (262h41m35.226245408s) [noise=true , address=145.79.227.40:4050 , piece_id=QPPTYML43XJRZCVQ5QKVTJMWOH4ETB6P2QUYQTPERBJ73OROLTWA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.355598334 (262h41m35.226681379s) [noise=true , address=145.79.227.51:4410 , piece_id=56VWGQ2SOLXWSNIXJMRI2VQ7VXYILL5EZN45IPSK6ZCWEJA4U2RA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.381079334 (262h41m35.252162369s) [noise=true , address=145.79.227.23:4545 , piece_id=PNRKCEAPMKBQTQMP7OOTS3S5MEW74VFREWCHZ5XAC7IDWF6S5MWA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.389735517 (262h41m35.260818552s) [noise=true , address=145.79.227.38:4979 , piece_id=4NOR7PSPCHWIKHA5GGGPJA3FXNCNEXKWKOLBMORBEZG4NKXRTR4Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.476608340 (262h41m35.347691385s) [noise=true , address=45.91.116.14:4192 , piece_id=WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.495258663 (262h41m35.366341718s) [noise=true , address=45.91.116.6:4976 , piece_id=GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.495283441 (262h41m35.366366486s) [seg_pos={0 0}, upload=230290]: Segment reached optimal threshold of 13 pieces. Dec 16 14:10:35.495690136 (262h41m35.366773191s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 16 14:10:35.495700315 (262h41m35.36678335s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.495951893 (262h41m35.367034929s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 16 14:10:35.495960940 (262h41m35.367043975s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.496269408 (262h41m35.367352443s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: context canceled; context canceled Dec 16 14:10:35.496278666 (262h41m35.367361701s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.496319834 (262h41m35.36740287s) [seg_pos={0 0}, upload=230290]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:10:35.496422099 (262h41m35.367505144s) [upload=230290]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:10:35.496427990 (262h41m35.367511035s) [upload=230290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:10:35.815631686 (262h41m35.686714751s) [upload=230290]: Done uploading segments... Dec 16 14:10:35.815655911 (262h41m35.686738966s) [upload=230290]: Upload finished. err: <nil> Duration: 5.908449577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 55.240334402s trace: https://jaeger.dev.storj.io/trace/ba2c44dde12044 FAIL (2025-12-16 14:14:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=229725]: Uploading piece... Dec 16 13:30:17.600801266 (262h1m17.471884321s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.600906235 (262h1m17.471989281s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601269747 (262h1m17.472352782s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601276790 (262h1m17.472359835s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601630013 (262h1m17.472713048s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601640714 (262h1m17.472723749s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601965091 (262h1m17.473048126s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602047147 (262h1m17.473130192s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602246808 (262h1m17.473329853s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602601232 (262h1m17.473684277s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602716132 (262h1m17.473799167s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602996997 (262h1m17.474080042s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603027615 (262h1m17.47411065s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603255378 (262h1m17.474338413s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603501387 (262h1m17.474584432s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603959158 (262h1m17.475042203s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.604485590 (262h1m17.475568635s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:19.111089229 (262h1m18.982172264s) [upload=229725]: Next returned nil segment Dec 16 13:30:19.111104708 (262h1m18.982187753s) [upload=229725]: Waiting for error group managing segments... Dec 16 13:30:19.527483020 (262h1m19.398566075s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.621753699 (262h1m19.492836744s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.670564204 (262h1m19.541647249s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.672302895 (262h1m19.54338594s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.716239789 (262h1m19.587322834s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.735358003 (262h1m19.606441048s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.750731367 (262h1m19.621814412s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.764604575 (262h1m19.63568761s) [noise=true , address=45.91.116.7:4029 , piece_id=NZ5HRBRJ3QMNZMXJLKD5NPC3ZZFAS7VIDQNXPURRPWOCMPTXFUNQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.796960486 (262h1m19.668043531s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.812007538 (262h1m19.683090573s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.032271213 (262h1m19.903354248s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.099742841 (262h1m19.970825886s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.231931353 (262h1m20.103014398s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394678143 (262h1m20.265761188s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394706487 (262h1m20.265789532s) [seg_pos={0 0}, upload=229725]: Segment reached optimal threshold of 14 pieces. Dec 16 13:30:20.394971411 (262h1m20.266054466s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.394981671 (262h1m20.266064716s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395367956 (262h1m20.266451001s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395378747 (262h1m20.266461792s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395542348 (262h1m20.266625393s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395565903 (262h1m20.266648948s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395688968 (262h1m20.266772013s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395695741 (262h1m20.266778786s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395718144 (262h1m20.266801189s) [seg_pos={0 0}, upload=229725]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 13:30:20.395862569 (262h1m20.266945604s) [upload=229725]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:30:20.395896854 (262h1m20.266979899s) [upload=229725]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:30:20.679088774 (262h1m20.550171819s) [upload=229725]: Done uploading segments... Dec 16 13:30:20.679109594 (262h1m20.550192629s) [upload=229725]: Upload finished. err: <nil> Duration: 55.240357286s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................X.XX....X.................] duration: 25.37318474s trace: https://jaeger.dev.storj.io/trace/24f03ce3e3f5cb2d FAIL (2025-12-16 14:14:22 UTC) orbiter warning: context deadline exceeded Duration: 25.373249264s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X....X......X.....] duration: 6.322262487s trace: https://jaeger.dev.storj.io/trace/e49c6365710000b FAIL (2025-12-16 14:14:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ) context deadline exceeded ======LOGS====== Dec 16 14:10:32.985073926 (262h41m32.856156961s) [upload=230290]: Starting upload Dec 16 14:10:32.985154330 (262h41m32.856237375s) [upload=230290]: Done starting upload Dec 16 14:10:32.985233330 (262h41m32.856316365s) [upload=230290]: Uploading segments... Dec 16 14:10:32.985485239 (262h41m32.856568294s) [upload=230290]: Got next segment. Inline: false Dec 16 14:10:32.985601450 (262h41m32.856684485s) [upload=230290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:10:32.985608184 (262h41m32.856691229s) [upload=230290]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:10:33.255392794 (262h41m33.126475829s) [seg_pos={0 0}, upload=230290]: Begin upload segment... Dec 16 14:10:33.255499337 (262h41m33.126582372s) [seg_pos={0 0}, upload=230290]: Done begin upload segment. Dec 16 14:10:33.255557117 (262h41m33.126640152s) [noise=true , address=145.79.227.51:4410 , piece_id=56VWGQ2SOLXWSNIXJMRI2VQ7VXYILL5EZN45IPSK6ZCWEJA4U2RA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.255585682 (262h41m33.126668727s) [noise=true , address=145.79.227.23:4545 , piece_id=PNRKCEAPMKBQTQMP7OOTS3S5MEW74VFREWCHZ5XAC7IDWF6S5MWA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.256346882 (262h41m33.127429917s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.256562532 (262h41m33.127645577s) [noise=true , address=145.79.227.40:4050 , piece_id=QPPTYML43XJRZCVQ5QKVTJMWOH4ETB6P2QUYQTPERBJ73OROLTWA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.257542648 (262h41m33.128625693s) [noise=true , address=45.91.116.6:4976 , piece_id=GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.258212524 (262h41m33.12929556s) [noise=true , address=145.79.227.17:4972 , piece_id=SS2ZYEXA2OWPY4MXJIT53HXHT5BZ5FBNFNECV3JVIPRTZCWRLXHQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.258731993 (262h41m33.129815028s) [noise=true , address=145.79.227.9:4365 , piece_id=7CHLK56ZSQ3VMLCHTGWEPXTYSOI7L6IX5IK7UUTFPYK434Y2BJ5Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259079935 (262h41m33.13016297s) [noise=true , address=145.79.227.45:4231 , piece_id=GI24XZVNFRHGDWBWFE3KE7W5OHBVEHLJ3B2BMCP6PNKGWJRAAYMA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259188884 (262h41m33.130271919s) [noise=true , address=45.91.116.14:4192 , piece_id=WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259521847 (262h41m33.130604892s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259824013 (262h41m33.130907048s) [noise=true , address=145.79.227.38:4979 , piece_id=4NOR7PSPCHWIKHA5GGGPJA3FXNCNEXKWKOLBMORBEZG4NKXRTR4Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260168439 (262h41m33.131251484s) [noise=true , address=145.79.227.32:4836 , piece_id=F6L5YJ773REOLNMASD2JD7BTGFKB2UBXZMBPYERZ5I6DTYY74OMA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260470092 (262h41m33.131553137s) [noise=true , address=145.79.227.15:4876 , piece_id=DACMMDUBQG2TF2VERLEJ5CJXEWKGYTYTQ376KGICJNMCFFG4RWEA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260751117 (262h41m33.131834163s) [noise=true , address=145.79.227.50:4374 , piece_id=O2T3JYB2C6RRAB3NXMQHBXF4EEGQES2KYPBGR6JM6SIIGFYNWPZQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.261047001 (262h41m33.132130036s) [noise=true , address=145.79.227.47:4267 , piece_id=CRCIH27Q7AYOJED5Q5Y3LB4GXOWJNEVAMXOGBTHEF6WKBGVW2EHQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.261378823 (262h41m33.132461858s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:34.973936146 (262h41m34.845019181s) [upload=230290]: Next returned nil segment Dec 16 14:10:34.973950845 (262h41m34.84503388s) [upload=230290]: Waiting for error group managing segments... Dec 16 14:10:35.255716630 (262h41m35.126799675s) [noise=true , address=145.79.227.45:4231 , piece_id=GI24XZVNFRHGDWBWFE3KE7W5OHBVEHLJ3B2BMCP6PNKGWJRAAYMA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.260286111 (262h41m35.131369156s) [noise=true , address=145.79.227.47:4267 , piece_id=CRCIH27Q7AYOJED5Q5Y3LB4GXOWJNEVAMXOGBTHEF6WKBGVW2EHQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.266206785 (262h41m35.13728982s) [noise=true , address=145.79.227.17:4972 , piece_id=SS2ZYEXA2OWPY4MXJIT53HXHT5BZ5FBNFNECV3JVIPRTZCWRLXHQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.269766433 (262h41m35.140849468s) [noise=true , address=145.79.227.15:4876 , piece_id=DACMMDUBQG2TF2VERLEJ5CJXEWKGYTYTQ376KGICJNMCFFG4RWEA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.277099697 (262h41m35.148182742s) [noise=true , address=145.79.227.50:4374 , piece_id=O2T3JYB2C6RRAB3NXMQHBXF4EEGQES2KYPBGR6JM6SIIGFYNWPZQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.277901374 (262h41m35.148984409s) [noise=true , address=145.79.227.32:4836 , piece_id=F6L5YJ773REOLNMASD2JD7BTGFKB2UBXZMBPYERZ5I6DTYY74OMA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.335067849 (262h41m35.206150884s) [noise=true , address=145.79.227.9:4365 , piece_id=7CHLK56ZSQ3VMLCHTGWEPXTYSOI7L6IX5IK7UUTFPYK434Y2BJ5Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.355162373 (262h41m35.226245408s) [noise=true , address=145.79.227.40:4050 , piece_id=QPPTYML43XJRZCVQ5QKVTJMWOH4ETB6P2QUYQTPERBJ73OROLTWA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.355598334 (262h41m35.226681379s) [noise=true , address=145.79.227.51:4410 , piece_id=56VWGQ2SOLXWSNIXJMRI2VQ7VXYILL5EZN45IPSK6ZCWEJA4U2RA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.381079334 (262h41m35.252162369s) [noise=true , address=145.79.227.23:4545 , piece_id=PNRKCEAPMKBQTQMP7OOTS3S5MEW74VFREWCHZ5XAC7IDWF6S5MWA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.389735517 (262h41m35.260818552s) [noise=true , address=145.79.227.38:4979 , piece_id=4NOR7PSPCHWIKHA5GGGPJA3FXNCNEXKWKOLBMORBEZG4NKXRTR4Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.476608340 (262h41m35.347691385s) [noise=true , address=45.91.116.14:4192 , piece_id=WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.495258663 (262h41m35.366341718s) [noise=true , address=45.91.116.6:4976 , piece_id=GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.495283441 (262h41m35.366366486s) [seg_pos={0 0}, upload=230290]: Segment reached optimal threshold of 13 pieces. Dec 16 14:10:35.495690136 (262h41m35.366773191s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 16 14:10:35.495700315 (262h41m35.36678335s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.495951893 (262h41m35.367034929s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 16 14:10:35.495960940 (262h41m35.367043975s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.496269408 (262h41m35.367352443s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: context canceled; context canceled Dec 16 14:10:35.496278666 (262h41m35.367361701s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.496319834 (262h41m35.36740287s) [seg_pos={0 0}, upload=230290]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:10:35.496422099 (262h41m35.367505144s) [upload=230290]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:10:35.496427990 (262h41m35.367511035s) [upload=230290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:10:35.815631686 (262h41m35.686714751s) [upload=230290]: Done uploading segments... Dec 16 14:10:35.815655911 (262h41m35.686738966s) [upload=230290]: Upload finished. err: <nil> Duration: 6.322291763s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [.X.X.XXX.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XXX] duration: 5.02176295s trace: https://jaeger.dev.storj.io/trace/ada29348c12f764 FAIL (2025-12-16 14:13:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.021807224s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [X.X.X.XXX.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.XX] duration: 5.001699414s trace: https://jaeger.dev.storj.io/trace/41af22cc1407f6a7 FAIL (2025-12-16 14:13:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001723449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 49.121035403s trace: https://jaeger.dev.storj.io/trace/ba2c44dde12044 FAIL (2025-12-16 14:13:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=229725]: Uploading piece... Dec 16 13:30:17.600801266 (262h1m17.471884321s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.600906235 (262h1m17.471989281s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601269747 (262h1m17.472352782s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601276790 (262h1m17.472359835s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601630013 (262h1m17.472713048s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601640714 (262h1m17.472723749s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601965091 (262h1m17.473048126s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602047147 (262h1m17.473130192s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602246808 (262h1m17.473329853s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602601232 (262h1m17.473684277s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602716132 (262h1m17.473799167s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602996997 (262h1m17.474080042s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603027615 (262h1m17.47411065s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603255378 (262h1m17.474338413s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603501387 (262h1m17.474584432s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603959158 (262h1m17.475042203s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.604485590 (262h1m17.475568635s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:19.111089229 (262h1m18.982172264s) [upload=229725]: Next returned nil segment Dec 16 13:30:19.111104708 (262h1m18.982187753s) [upload=229725]: Waiting for error group managing segments... Dec 16 13:30:19.527483020 (262h1m19.398566075s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.621753699 (262h1m19.492836744s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.670564204 (262h1m19.541647249s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.672302895 (262h1m19.54338594s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.716239789 (262h1m19.587322834s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.735358003 (262h1m19.606441048s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.750731367 (262h1m19.621814412s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.764604575 (262h1m19.63568761s) [noise=true , address=45.91.116.7:4029 , piece_id=NZ5HRBRJ3QMNZMXJLKD5NPC3ZZFAS7VIDQNXPURRPWOCMPTXFUNQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.796960486 (262h1m19.668043531s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.812007538 (262h1m19.683090573s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.032271213 (262h1m19.903354248s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.099742841 (262h1m19.970825886s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.231931353 (262h1m20.103014398s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394678143 (262h1m20.265761188s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394706487 (262h1m20.265789532s) [seg_pos={0 0}, upload=229725]: Segment reached optimal threshold of 14 pieces. Dec 16 13:30:20.394971411 (262h1m20.266054466s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.394981671 (262h1m20.266064716s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395367956 (262h1m20.266451001s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395378747 (262h1m20.266461792s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395542348 (262h1m20.266625393s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395565903 (262h1m20.266648948s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395688968 (262h1m20.266772013s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395695741 (262h1m20.266778786s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395718144 (262h1m20.266801189s) [seg_pos={0 0}, upload=229725]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 13:30:20.395862569 (262h1m20.266945604s) [upload=229725]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:30:20.395896854 (262h1m20.266979899s) [upload=229725]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:30:20.679088774 (262h1m20.550171819s) [upload=229725]: Done uploading segments... Dec 16 13:30:20.679109594 (262h1m20.550192629s) [upload=229725]: Upload finished. err: <nil> Duration: 49.121054559s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X.XX....X................] duration: 17.381443255s trace: https://jaeger.dev.storj.io/trace/24f03ce3e3f5cb2d FAIL (2025-12-16 14:13:09 UTC) orbiter warning: context deadline exceeded Duration: 17.381459526s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X....X......X....] duration: 6.222108085s trace: https://jaeger.dev.storj.io/trace/e49c6365710000b FAIL (2025-12-16 14:12:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ) context deadline exceeded ======LOGS====== Dec 16 14:10:32.985073926 (262h41m32.856156961s) [upload=230290]: Starting upload Dec 16 14:10:32.985154330 (262h41m32.856237375s) [upload=230290]: Done starting upload Dec 16 14:10:32.985233330 (262h41m32.856316365s) [upload=230290]: Uploading segments... Dec 16 14:10:32.985485239 (262h41m32.856568294s) [upload=230290]: Got next segment. Inline: false Dec 16 14:10:32.985601450 (262h41m32.856684485s) [upload=230290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:10:32.985608184 (262h41m32.856691229s) [upload=230290]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:10:33.255392794 (262h41m33.126475829s) [seg_pos={0 0}, upload=230290]: Begin upload segment... Dec 16 14:10:33.255499337 (262h41m33.126582372s) [seg_pos={0 0}, upload=230290]: Done begin upload segment. Dec 16 14:10:33.255557117 (262h41m33.126640152s) [noise=true , address=145.79.227.51:4410 , piece_id=56VWGQ2SOLXWSNIXJMRI2VQ7VXYILL5EZN45IPSK6ZCWEJA4U2RA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.255585682 (262h41m33.126668727s) [noise=true , address=145.79.227.23:4545 , piece_id=PNRKCEAPMKBQTQMP7OOTS3S5MEW74VFREWCHZ5XAC7IDWF6S5MWA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.256346882 (262h41m33.127429917s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.256562532 (262h41m33.127645577s) [noise=true , address=145.79.227.40:4050 , piece_id=QPPTYML43XJRZCVQ5QKVTJMWOH4ETB6P2QUYQTPERBJ73OROLTWA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.257542648 (262h41m33.128625693s) [noise=true , address=45.91.116.6:4976 , piece_id=GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.258212524 (262h41m33.12929556s) [noise=true , address=145.79.227.17:4972 , piece_id=SS2ZYEXA2OWPY4MXJIT53HXHT5BZ5FBNFNECV3JVIPRTZCWRLXHQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.258731993 (262h41m33.129815028s) [noise=true , address=145.79.227.9:4365 , piece_id=7CHLK56ZSQ3VMLCHTGWEPXTYSOI7L6IX5IK7UUTFPYK434Y2BJ5Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259079935 (262h41m33.13016297s) [noise=true , address=145.79.227.45:4231 , piece_id=GI24XZVNFRHGDWBWFE3KE7W5OHBVEHLJ3B2BMCP6PNKGWJRAAYMA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259188884 (262h41m33.130271919s) [noise=true , address=45.91.116.14:4192 , piece_id=WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259521847 (262h41m33.130604892s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259824013 (262h41m33.130907048s) [noise=true , address=145.79.227.38:4979 , piece_id=4NOR7PSPCHWIKHA5GGGPJA3FXNCNEXKWKOLBMORBEZG4NKXRTR4Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260168439 (262h41m33.131251484s) [noise=true , address=145.79.227.32:4836 , piece_id=F6L5YJ773REOLNMASD2JD7BTGFKB2UBXZMBPYERZ5I6DTYY74OMA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260470092 (262h41m33.131553137s) [noise=true , address=145.79.227.15:4876 , piece_id=DACMMDUBQG2TF2VERLEJ5CJXEWKGYTYTQ376KGICJNMCFFG4RWEA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260751117 (262h41m33.131834163s) [noise=true , address=145.79.227.50:4374 , piece_id=O2T3JYB2C6RRAB3NXMQHBXF4EEGQES2KYPBGR6JM6SIIGFYNWPZQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.261047001 (262h41m33.132130036s) [noise=true , address=145.79.227.47:4267 , piece_id=CRCIH27Q7AYOJED5Q5Y3LB4GXOWJNEVAMXOGBTHEF6WKBGVW2EHQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.261378823 (262h41m33.132461858s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:34.973936146 (262h41m34.845019181s) [upload=230290]: Next returned nil segment Dec 16 14:10:34.973950845 (262h41m34.84503388s) [upload=230290]: Waiting for error group managing segments... Dec 16 14:10:35.255716630 (262h41m35.126799675s) [noise=true , address=145.79.227.45:4231 , piece_id=GI24XZVNFRHGDWBWFE3KE7W5OHBVEHLJ3B2BMCP6PNKGWJRAAYMA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.260286111 (262h41m35.131369156s) [noise=true , address=145.79.227.47:4267 , piece_id=CRCIH27Q7AYOJED5Q5Y3LB4GXOWJNEVAMXOGBTHEF6WKBGVW2EHQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.266206785 (262h41m35.13728982s) [noise=true , address=145.79.227.17:4972 , piece_id=SS2ZYEXA2OWPY4MXJIT53HXHT5BZ5FBNFNECV3JVIPRTZCWRLXHQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.269766433 (262h41m35.140849468s) [noise=true , address=145.79.227.15:4876 , piece_id=DACMMDUBQG2TF2VERLEJ5CJXEWKGYTYTQ376KGICJNMCFFG4RWEA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.277099697 (262h41m35.148182742s) [noise=true , address=145.79.227.50:4374 , piece_id=O2T3JYB2C6RRAB3NXMQHBXF4EEGQES2KYPBGR6JM6SIIGFYNWPZQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.277901374 (262h41m35.148984409s) [noise=true , address=145.79.227.32:4836 , piece_id=F6L5YJ773REOLNMASD2JD7BTGFKB2UBXZMBPYERZ5I6DTYY74OMA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.335067849 (262h41m35.206150884s) [noise=true , address=145.79.227.9:4365 , piece_id=7CHLK56ZSQ3VMLCHTGWEPXTYSOI7L6IX5IK7UUTFPYK434Y2BJ5Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.355162373 (262h41m35.226245408s) [noise=true , address=145.79.227.40:4050 , piece_id=QPPTYML43XJRZCVQ5QKVTJMWOH4ETB6P2QUYQTPERBJ73OROLTWA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.355598334 (262h41m35.226681379s) [noise=true , address=145.79.227.51:4410 , piece_id=56VWGQ2SOLXWSNIXJMRI2VQ7VXYILL5EZN45IPSK6ZCWEJA4U2RA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.381079334 (262h41m35.252162369s) [noise=true , address=145.79.227.23:4545 , piece_id=PNRKCEAPMKBQTQMP7OOTS3S5MEW74VFREWCHZ5XAC7IDWF6S5MWA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.389735517 (262h41m35.260818552s) [noise=true , address=145.79.227.38:4979 , piece_id=4NOR7PSPCHWIKHA5GGGPJA3FXNCNEXKWKOLBMORBEZG4NKXRTR4Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.476608340 (262h41m35.347691385s) [noise=true , address=45.91.116.14:4192 , piece_id=WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.495258663 (262h41m35.366341718s) [noise=true , address=45.91.116.6:4976 , piece_id=GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.495283441 (262h41m35.366366486s) [seg_pos={0 0}, upload=230290]: Segment reached optimal threshold of 13 pieces. Dec 16 14:10:35.495690136 (262h41m35.366773191s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 16 14:10:35.495700315 (262h41m35.36678335s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.495951893 (262h41m35.367034929s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 16 14:10:35.495960940 (262h41m35.367043975s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.496269408 (262h41m35.367352443s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: context canceled; context canceled Dec 16 14:10:35.496278666 (262h41m35.367361701s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.496319834 (262h41m35.36740287s) [seg_pos={0 0}, upload=230290]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:10:35.496422099 (262h41m35.367505144s) [upload=230290]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:10:35.496427990 (262h41m35.367511035s) [upload=230290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:10:35.815631686 (262h41m35.686714751s) [upload=230290]: Done uploading segments... Dec 16 14:10:35.815655911 (262h41m35.686738966s) [upload=230290]: Upload finished. err: <nil> Duration: 6.222134355s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................X.XX....X...............] duration: 14.77905505s trace: https://jaeger.dev.storj.io/trace/24f03ce3e3f5cb2d FAIL (2025-12-16 14:12:44 UTC) orbiter warning: context deadline exceeded Duration: 14.779082811s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 48.06195952s trace: https://jaeger.dev.storj.io/trace/ba2c44dde12044 FAIL (2025-12-16 14:12:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=229725]: Uploading piece... Dec 16 13:30:17.600801266 (262h1m17.471884321s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.600906235 (262h1m17.471989281s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601269747 (262h1m17.472352782s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601276790 (262h1m17.472359835s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601630013 (262h1m17.472713048s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601640714 (262h1m17.472723749s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601965091 (262h1m17.473048126s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602047147 (262h1m17.473130192s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602246808 (262h1m17.473329853s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602601232 (262h1m17.473684277s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602716132 (262h1m17.473799167s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602996997 (262h1m17.474080042s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603027615 (262h1m17.47411065s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603255378 (262h1m17.474338413s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603501387 (262h1m17.474584432s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603959158 (262h1m17.475042203s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.604485590 (262h1m17.475568635s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:19.111089229 (262h1m18.982172264s) [upload=229725]: Next returned nil segment Dec 16 13:30:19.111104708 (262h1m18.982187753s) [upload=229725]: Waiting for error group managing segments... Dec 16 13:30:19.527483020 (262h1m19.398566075s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.621753699 (262h1m19.492836744s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.670564204 (262h1m19.541647249s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.672302895 (262h1m19.54338594s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.716239789 (262h1m19.587322834s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.735358003 (262h1m19.606441048s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.750731367 (262h1m19.621814412s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.764604575 (262h1m19.63568761s) [noise=true , address=45.91.116.7:4029 , piece_id=NZ5HRBRJ3QMNZMXJLKD5NPC3ZZFAS7VIDQNXPURRPWOCMPTXFUNQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.796960486 (262h1m19.668043531s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.812007538 (262h1m19.683090573s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.032271213 (262h1m19.903354248s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.099742841 (262h1m19.970825886s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.231931353 (262h1m20.103014398s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394678143 (262h1m20.265761188s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394706487 (262h1m20.265789532s) [seg_pos={0 0}, upload=229725]: Segment reached optimal threshold of 14 pieces. Dec 16 13:30:20.394971411 (262h1m20.266054466s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.394981671 (262h1m20.266064716s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395367956 (262h1m20.266451001s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395378747 (262h1m20.266461792s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395542348 (262h1m20.266625393s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395565903 (262h1m20.266648948s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395688968 (262h1m20.266772013s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395695741 (262h1m20.266778786s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395718144 (262h1m20.266801189s) [seg_pos={0 0}, upload=229725]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 13:30:20.395862569 (262h1m20.266945604s) [upload=229725]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:30:20.395896854 (262h1m20.266979899s) [upload=229725]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:30:20.679088774 (262h1m20.550171819s) [upload=229725]: Done uploading segments... Dec 16 13:30:20.679109594 (262h1m20.550192629s) [upload=229725]: Upload finished. err: <nil> Duration: 48.061988967s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XX.X.X.XXX.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.X] duration: 5.002525758s trace: https://jaeger.dev.storj.io/trace/37e1d8ada46626d8 FAIL (2025-12-16 14:12:19 UTC) orbiter warning: context deadline exceeded Duration: 5.002543362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X....X......X...] duration: 5.883271522s trace: https://jaeger.dev.storj.io/trace/e49c6365710000b FAIL (2025-12-16 14:12:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ) context deadline exceeded ======LOGS====== Dec 16 14:10:32.985073926 (262h41m32.856156961s) [upload=230290]: Starting upload Dec 16 14:10:32.985154330 (262h41m32.856237375s) [upload=230290]: Done starting upload Dec 16 14:10:32.985233330 (262h41m32.856316365s) [upload=230290]: Uploading segments... Dec 16 14:10:32.985485239 (262h41m32.856568294s) [upload=230290]: Got next segment. Inline: false Dec 16 14:10:32.985601450 (262h41m32.856684485s) [upload=230290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:10:32.985608184 (262h41m32.856691229s) [upload=230290]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:10:33.255392794 (262h41m33.126475829s) [seg_pos={0 0}, upload=230290]: Begin upload segment... Dec 16 14:10:33.255499337 (262h41m33.126582372s) [seg_pos={0 0}, upload=230290]: Done begin upload segment. Dec 16 14:10:33.255557117 (262h41m33.126640152s) [noise=true , address=145.79.227.51:4410 , piece_id=56VWGQ2SOLXWSNIXJMRI2VQ7VXYILL5EZN45IPSK6ZCWEJA4U2RA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.255585682 (262h41m33.126668727s) [noise=true , address=145.79.227.23:4545 , piece_id=PNRKCEAPMKBQTQMP7OOTS3S5MEW74VFREWCHZ5XAC7IDWF6S5MWA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.256346882 (262h41m33.127429917s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.256562532 (262h41m33.127645577s) [noise=true , address=145.79.227.40:4050 , piece_id=QPPTYML43XJRZCVQ5QKVTJMWOH4ETB6P2QUYQTPERBJ73OROLTWA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.257542648 (262h41m33.128625693s) [noise=true , address=45.91.116.6:4976 , piece_id=GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.258212524 (262h41m33.12929556s) [noise=true , address=145.79.227.17:4972 , piece_id=SS2ZYEXA2OWPY4MXJIT53HXHT5BZ5FBNFNECV3JVIPRTZCWRLXHQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.258731993 (262h41m33.129815028s) [noise=true , address=145.79.227.9:4365 , piece_id=7CHLK56ZSQ3VMLCHTGWEPXTYSOI7L6IX5IK7UUTFPYK434Y2BJ5Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259079935 (262h41m33.13016297s) [noise=true , address=145.79.227.45:4231 , piece_id=GI24XZVNFRHGDWBWFE3KE7W5OHBVEHLJ3B2BMCP6PNKGWJRAAYMA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259188884 (262h41m33.130271919s) [noise=true , address=45.91.116.14:4192 , piece_id=WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259521847 (262h41m33.130604892s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259824013 (262h41m33.130907048s) [noise=true , address=145.79.227.38:4979 , piece_id=4NOR7PSPCHWIKHA5GGGPJA3FXNCNEXKWKOLBMORBEZG4NKXRTR4Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260168439 (262h41m33.131251484s) [noise=true , address=145.79.227.32:4836 , piece_id=F6L5YJ773REOLNMASD2JD7BTGFKB2UBXZMBPYERZ5I6DTYY74OMA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260470092 (262h41m33.131553137s) [noise=true , address=145.79.227.15:4876 , piece_id=DACMMDUBQG2TF2VERLEJ5CJXEWKGYTYTQ376KGICJNMCFFG4RWEA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260751117 (262h41m33.131834163s) [noise=true , address=145.79.227.50:4374 , piece_id=O2T3JYB2C6RRAB3NXMQHBXF4EEGQES2KYPBGR6JM6SIIGFYNWPZQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.261047001 (262h41m33.132130036s) [noise=true , address=145.79.227.47:4267 , piece_id=CRCIH27Q7AYOJED5Q5Y3LB4GXOWJNEVAMXOGBTHEF6WKBGVW2EHQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.261378823 (262h41m33.132461858s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:34.973936146 (262h41m34.845019181s) [upload=230290]: Next returned nil segment Dec 16 14:10:34.973950845 (262h41m34.84503388s) [upload=230290]: Waiting for error group managing segments... Dec 16 14:10:35.255716630 (262h41m35.126799675s) [noise=true , address=145.79.227.45:4231 , piece_id=GI24XZVNFRHGDWBWFE3KE7W5OHBVEHLJ3B2BMCP6PNKGWJRAAYMA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.260286111 (262h41m35.131369156s) [noise=true , address=145.79.227.47:4267 , piece_id=CRCIH27Q7AYOJED5Q5Y3LB4GXOWJNEVAMXOGBTHEF6WKBGVW2EHQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.266206785 (262h41m35.13728982s) [noise=true , address=145.79.227.17:4972 , piece_id=SS2ZYEXA2OWPY4MXJIT53HXHT5BZ5FBNFNECV3JVIPRTZCWRLXHQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.269766433 (262h41m35.140849468s) [noise=true , address=145.79.227.15:4876 , piece_id=DACMMDUBQG2TF2VERLEJ5CJXEWKGYTYTQ376KGICJNMCFFG4RWEA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.277099697 (262h41m35.148182742s) [noise=true , address=145.79.227.50:4374 , piece_id=O2T3JYB2C6RRAB3NXMQHBXF4EEGQES2KYPBGR6JM6SIIGFYNWPZQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.277901374 (262h41m35.148984409s) [noise=true , address=145.79.227.32:4836 , piece_id=F6L5YJ773REOLNMASD2JD7BTGFKB2UBXZMBPYERZ5I6DTYY74OMA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.335067849 (262h41m35.206150884s) [noise=true , address=145.79.227.9:4365 , piece_id=7CHLK56ZSQ3VMLCHTGWEPXTYSOI7L6IX5IK7UUTFPYK434Y2BJ5Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.355162373 (262h41m35.226245408s) [noise=true , address=145.79.227.40:4050 , piece_id=QPPTYML43XJRZCVQ5QKVTJMWOH4ETB6P2QUYQTPERBJ73OROLTWA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.355598334 (262h41m35.226681379s) [noise=true , address=145.79.227.51:4410 , piece_id=56VWGQ2SOLXWSNIXJMRI2VQ7VXYILL5EZN45IPSK6ZCWEJA4U2RA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.381079334 (262h41m35.252162369s) [noise=true , address=145.79.227.23:4545 , piece_id=PNRKCEAPMKBQTQMP7OOTS3S5MEW74VFREWCHZ5XAC7IDWF6S5MWA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.389735517 (262h41m35.260818552s) [noise=true , address=145.79.227.38:4979 , piece_id=4NOR7PSPCHWIKHA5GGGPJA3FXNCNEXKWKOLBMORBEZG4NKXRTR4Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.476608340 (262h41m35.347691385s) [noise=true , address=45.91.116.14:4192 , piece_id=WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.495258663 (262h41m35.366341718s) [noise=true , address=45.91.116.6:4976 , piece_id=GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.495283441 (262h41m35.366366486s) [seg_pos={0 0}, upload=230290]: Segment reached optimal threshold of 13 pieces. Dec 16 14:10:35.495690136 (262h41m35.366773191s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 16 14:10:35.495700315 (262h41m35.36678335s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.495951893 (262h41m35.367034929s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 16 14:10:35.495960940 (262h41m35.367043975s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.496269408 (262h41m35.367352443s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: context canceled; context canceled Dec 16 14:10:35.496278666 (262h41m35.367361701s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.496319834 (262h41m35.36740287s) [seg_pos={0 0}, upload=230290]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:10:35.496422099 (262h41m35.367505144s) [upload=230290]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:10:35.496427990 (262h41m35.367511035s) [upload=230290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:10:35.815631686 (262h41m35.686714751s) [upload=230290]: Done uploading segments... Dec 16 14:10:35.815655911 (262h41m35.686738966s) [upload=230290]: Upload finished. err: <nil> Duration: 5.883289166s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X.XX....X..............] duration: 21.112847092s trace: https://jaeger.dev.storj.io/trace/24f03ce3e3f5cb2d FAIL (2025-12-16 14:11:56 UTC) orbiter warning: context deadline exceeded Duration: 21.112873332s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X....X......X..] duration: 5.95907581s trace: https://jaeger.dev.storj.io/trace/e49c6365710000b FAIL (2025-12-16 14:11:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ) context deadline exceeded ======LOGS====== Dec 16 14:10:32.985073926 (262h41m32.856156961s) [upload=230290]: Starting upload Dec 16 14:10:32.985154330 (262h41m32.856237375s) [upload=230290]: Done starting upload Dec 16 14:10:32.985233330 (262h41m32.856316365s) [upload=230290]: Uploading segments... Dec 16 14:10:32.985485239 (262h41m32.856568294s) [upload=230290]: Got next segment. Inline: false Dec 16 14:10:32.985601450 (262h41m32.856684485s) [upload=230290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:10:32.985608184 (262h41m32.856691229s) [upload=230290]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:10:33.255392794 (262h41m33.126475829s) [seg_pos={0 0}, upload=230290]: Begin upload segment... Dec 16 14:10:33.255499337 (262h41m33.126582372s) [seg_pos={0 0}, upload=230290]: Done begin upload segment. Dec 16 14:10:33.255557117 (262h41m33.126640152s) [noise=true , address=145.79.227.51:4410 , piece_id=56VWGQ2SOLXWSNIXJMRI2VQ7VXYILL5EZN45IPSK6ZCWEJA4U2RA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.255585682 (262h41m33.126668727s) [noise=true , address=145.79.227.23:4545 , piece_id=PNRKCEAPMKBQTQMP7OOTS3S5MEW74VFREWCHZ5XAC7IDWF6S5MWA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.256346882 (262h41m33.127429917s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.256562532 (262h41m33.127645577s) [noise=true , address=145.79.227.40:4050 , piece_id=QPPTYML43XJRZCVQ5QKVTJMWOH4ETB6P2QUYQTPERBJ73OROLTWA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.257542648 (262h41m33.128625693s) [noise=true , address=45.91.116.6:4976 , piece_id=GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.258212524 (262h41m33.12929556s) [noise=true , address=145.79.227.17:4972 , piece_id=SS2ZYEXA2OWPY4MXJIT53HXHT5BZ5FBNFNECV3JVIPRTZCWRLXHQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.258731993 (262h41m33.129815028s) [noise=true , address=145.79.227.9:4365 , piece_id=7CHLK56ZSQ3VMLCHTGWEPXTYSOI7L6IX5IK7UUTFPYK434Y2BJ5Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259079935 (262h41m33.13016297s) [noise=true , address=145.79.227.45:4231 , piece_id=GI24XZVNFRHGDWBWFE3KE7W5OHBVEHLJ3B2BMCP6PNKGWJRAAYMA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259188884 (262h41m33.130271919s) [noise=true , address=45.91.116.14:4192 , piece_id=WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259521847 (262h41m33.130604892s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259824013 (262h41m33.130907048s) [noise=true , address=145.79.227.38:4979 , piece_id=4NOR7PSPCHWIKHA5GGGPJA3FXNCNEXKWKOLBMORBEZG4NKXRTR4Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260168439 (262h41m33.131251484s) [noise=true , address=145.79.227.32:4836 , piece_id=F6L5YJ773REOLNMASD2JD7BTGFKB2UBXZMBPYERZ5I6DTYY74OMA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260470092 (262h41m33.131553137s) [noise=true , address=145.79.227.15:4876 , piece_id=DACMMDUBQG2TF2VERLEJ5CJXEWKGYTYTQ376KGICJNMCFFG4RWEA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260751117 (262h41m33.131834163s) [noise=true , address=145.79.227.50:4374 , piece_id=O2T3JYB2C6RRAB3NXMQHBXF4EEGQES2KYPBGR6JM6SIIGFYNWPZQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.261047001 (262h41m33.132130036s) [noise=true , address=145.79.227.47:4267 , piece_id=CRCIH27Q7AYOJED5Q5Y3LB4GXOWJNEVAMXOGBTHEF6WKBGVW2EHQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.261378823 (262h41m33.132461858s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:34.973936146 (262h41m34.845019181s) [upload=230290]: Next returned nil segment Dec 16 14:10:34.973950845 (262h41m34.84503388s) [upload=230290]: Waiting for error group managing segments... Dec 16 14:10:35.255716630 (262h41m35.126799675s) [noise=true , address=145.79.227.45:4231 , piece_id=GI24XZVNFRHGDWBWFE3KE7W5OHBVEHLJ3B2BMCP6PNKGWJRAAYMA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.260286111 (262h41m35.131369156s) [noise=true , address=145.79.227.47:4267 , piece_id=CRCIH27Q7AYOJED5Q5Y3LB4GXOWJNEVAMXOGBTHEF6WKBGVW2EHQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.266206785 (262h41m35.13728982s) [noise=true , address=145.79.227.17:4972 , piece_id=SS2ZYEXA2OWPY4MXJIT53HXHT5BZ5FBNFNECV3JVIPRTZCWRLXHQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.269766433 (262h41m35.140849468s) [noise=true , address=145.79.227.15:4876 , piece_id=DACMMDUBQG2TF2VERLEJ5CJXEWKGYTYTQ376KGICJNMCFFG4RWEA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.277099697 (262h41m35.148182742s) [noise=true , address=145.79.227.50:4374 , piece_id=O2T3JYB2C6RRAB3NXMQHBXF4EEGQES2KYPBGR6JM6SIIGFYNWPZQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.277901374 (262h41m35.148984409s) [noise=true , address=145.79.227.32:4836 , piece_id=F6L5YJ773REOLNMASD2JD7BTGFKB2UBXZMBPYERZ5I6DTYY74OMA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.335067849 (262h41m35.206150884s) [noise=true , address=145.79.227.9:4365 , piece_id=7CHLK56ZSQ3VMLCHTGWEPXTYSOI7L6IX5IK7UUTFPYK434Y2BJ5Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.355162373 (262h41m35.226245408s) [noise=true , address=145.79.227.40:4050 , piece_id=QPPTYML43XJRZCVQ5QKVTJMWOH4ETB6P2QUYQTPERBJ73OROLTWA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.355598334 (262h41m35.226681379s) [noise=true , address=145.79.227.51:4410 , piece_id=56VWGQ2SOLXWSNIXJMRI2VQ7VXYILL5EZN45IPSK6ZCWEJA4U2RA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.381079334 (262h41m35.252162369s) [noise=true , address=145.79.227.23:4545 , piece_id=PNRKCEAPMKBQTQMP7OOTS3S5MEW74VFREWCHZ5XAC7IDWF6S5MWA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.389735517 (262h41m35.260818552s) [noise=true , address=145.79.227.38:4979 , piece_id=4NOR7PSPCHWIKHA5GGGPJA3FXNCNEXKWKOLBMORBEZG4NKXRTR4Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.476608340 (262h41m35.347691385s) [noise=true , address=45.91.116.14:4192 , piece_id=WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.495258663 (262h41m35.366341718s) [noise=true , address=45.91.116.6:4976 , piece_id=GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.495283441 (262h41m35.366366486s) [seg_pos={0 0}, upload=230290]: Segment reached optimal threshold of 13 pieces. Dec 16 14:10:35.495690136 (262h41m35.366773191s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 16 14:10:35.495700315 (262h41m35.36678335s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.495951893 (262h41m35.367034929s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 16 14:10:35.495960940 (262h41m35.367043975s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.496269408 (262h41m35.367352443s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: context canceled; context canceled Dec 16 14:10:35.496278666 (262h41m35.367361701s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.496319834 (262h41m35.36740287s) [seg_pos={0 0}, upload=230290]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:10:35.496422099 (262h41m35.367505144s) [upload=230290]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:10:35.496427990 (262h41m35.367511035s) [upload=230290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:10:35.815631686 (262h41m35.686714751s) [upload=230290]: Done uploading segments... Dec 16 14:10:35.815655911 (262h41m35.686738966s) [upload=230290]: Upload finished. err: <nil> Duration: 5.959109435s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXX.X.X.XXX.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX.] duration: 4.077989026s trace: https://jaeger.dev.storj.io/trace/4bd4daad74e9900c FAIL (2025-12-16 14:11:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.078019414s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X....X......X.] duration: 6.288316458s trace: https://jaeger.dev.storj.io/trace/e49c6365710000b FAIL (2025-12-16 14:11:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ) context deadline exceeded ======LOGS====== Dec 16 14:10:32.985073926 (262h41m32.856156961s) [upload=230290]: Starting upload Dec 16 14:10:32.985154330 (262h41m32.856237375s) [upload=230290]: Done starting upload Dec 16 14:10:32.985233330 (262h41m32.856316365s) [upload=230290]: Uploading segments... Dec 16 14:10:32.985485239 (262h41m32.856568294s) [upload=230290]: Got next segment. Inline: false Dec 16 14:10:32.985601450 (262h41m32.856684485s) [upload=230290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:10:32.985608184 (262h41m32.856691229s) [upload=230290]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:10:33.255392794 (262h41m33.126475829s) [seg_pos={0 0}, upload=230290]: Begin upload segment... Dec 16 14:10:33.255499337 (262h41m33.126582372s) [seg_pos={0 0}, upload=230290]: Done begin upload segment. Dec 16 14:10:33.255557117 (262h41m33.126640152s) [noise=true , address=145.79.227.51:4410 , piece_id=56VWGQ2SOLXWSNIXJMRI2VQ7VXYILL5EZN45IPSK6ZCWEJA4U2RA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.255585682 (262h41m33.126668727s) [noise=true , address=145.79.227.23:4545 , piece_id=PNRKCEAPMKBQTQMP7OOTS3S5MEW74VFREWCHZ5XAC7IDWF6S5MWA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.256346882 (262h41m33.127429917s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.256562532 (262h41m33.127645577s) [noise=true , address=145.79.227.40:4050 , piece_id=QPPTYML43XJRZCVQ5QKVTJMWOH4ETB6P2QUYQTPERBJ73OROLTWA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.257542648 (262h41m33.128625693s) [noise=true , address=45.91.116.6:4976 , piece_id=GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.258212524 (262h41m33.12929556s) [noise=true , address=145.79.227.17:4972 , piece_id=SS2ZYEXA2OWPY4MXJIT53HXHT5BZ5FBNFNECV3JVIPRTZCWRLXHQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.258731993 (262h41m33.129815028s) [noise=true , address=145.79.227.9:4365 , piece_id=7CHLK56ZSQ3VMLCHTGWEPXTYSOI7L6IX5IK7UUTFPYK434Y2BJ5Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259079935 (262h41m33.13016297s) [noise=true , address=145.79.227.45:4231 , piece_id=GI24XZVNFRHGDWBWFE3KE7W5OHBVEHLJ3B2BMCP6PNKGWJRAAYMA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259188884 (262h41m33.130271919s) [noise=true , address=45.91.116.14:4192 , piece_id=WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259521847 (262h41m33.130604892s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259824013 (262h41m33.130907048s) [noise=true , address=145.79.227.38:4979 , piece_id=4NOR7PSPCHWIKHA5GGGPJA3FXNCNEXKWKOLBMORBEZG4NKXRTR4Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260168439 (262h41m33.131251484s) [noise=true , address=145.79.227.32:4836 , piece_id=F6L5YJ773REOLNMASD2JD7BTGFKB2UBXZMBPYERZ5I6DTYY74OMA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260470092 (262h41m33.131553137s) [noise=true , address=145.79.227.15:4876 , piece_id=DACMMDUBQG2TF2VERLEJ5CJXEWKGYTYTQ376KGICJNMCFFG4RWEA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260751117 (262h41m33.131834163s) [noise=true , address=145.79.227.50:4374 , piece_id=O2T3JYB2C6RRAB3NXMQHBXF4EEGQES2KYPBGR6JM6SIIGFYNWPZQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.261047001 (262h41m33.132130036s) [noise=true , address=145.79.227.47:4267 , piece_id=CRCIH27Q7AYOJED5Q5Y3LB4GXOWJNEVAMXOGBTHEF6WKBGVW2EHQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.261378823 (262h41m33.132461858s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:34.973936146 (262h41m34.845019181s) [upload=230290]: Next returned nil segment Dec 16 14:10:34.973950845 (262h41m34.84503388s) [upload=230290]: Waiting for error group managing segments... Dec 16 14:10:35.255716630 (262h41m35.126799675s) [noise=true , address=145.79.227.45:4231 , piece_id=GI24XZVNFRHGDWBWFE3KE7W5OHBVEHLJ3B2BMCP6PNKGWJRAAYMA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.260286111 (262h41m35.131369156s) [noise=true , address=145.79.227.47:4267 , piece_id=CRCIH27Q7AYOJED5Q5Y3LB4GXOWJNEVAMXOGBTHEF6WKBGVW2EHQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.266206785 (262h41m35.13728982s) [noise=true , address=145.79.227.17:4972 , piece_id=SS2ZYEXA2OWPY4MXJIT53HXHT5BZ5FBNFNECV3JVIPRTZCWRLXHQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.269766433 (262h41m35.140849468s) [noise=true , address=145.79.227.15:4876 , piece_id=DACMMDUBQG2TF2VERLEJ5CJXEWKGYTYTQ376KGICJNMCFFG4RWEA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.277099697 (262h41m35.148182742s) [noise=true , address=145.79.227.50:4374 , piece_id=O2T3JYB2C6RRAB3NXMQHBXF4EEGQES2KYPBGR6JM6SIIGFYNWPZQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.277901374 (262h41m35.148984409s) [noise=true , address=145.79.227.32:4836 , piece_id=F6L5YJ773REOLNMASD2JD7BTGFKB2UBXZMBPYERZ5I6DTYY74OMA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.335067849 (262h41m35.206150884s) [noise=true , address=145.79.227.9:4365 , piece_id=7CHLK56ZSQ3VMLCHTGWEPXTYSOI7L6IX5IK7UUTFPYK434Y2BJ5Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.355162373 (262h41m35.226245408s) [noise=true , address=145.79.227.40:4050 , piece_id=QPPTYML43XJRZCVQ5QKVTJMWOH4ETB6P2QUYQTPERBJ73OROLTWA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.355598334 (262h41m35.226681379s) [noise=true , address=145.79.227.51:4410 , piece_id=56VWGQ2SOLXWSNIXJMRI2VQ7VXYILL5EZN45IPSK6ZCWEJA4U2RA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.381079334 (262h41m35.252162369s) [noise=true , address=145.79.227.23:4545 , piece_id=PNRKCEAPMKBQTQMP7OOTS3S5MEW74VFREWCHZ5XAC7IDWF6S5MWA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.389735517 (262h41m35.260818552s) [noise=true , address=145.79.227.38:4979 , piece_id=4NOR7PSPCHWIKHA5GGGPJA3FXNCNEXKWKOLBMORBEZG4NKXRTR4Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.476608340 (262h41m35.347691385s) [noise=true , address=45.91.116.14:4192 , piece_id=WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.495258663 (262h41m35.366341718s) [noise=true , address=45.91.116.6:4976 , piece_id=GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.495283441 (262h41m35.366366486s) [seg_pos={0 0}, upload=230290]: Segment reached optimal threshold of 13 pieces. Dec 16 14:10:35.495690136 (262h41m35.366773191s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 16 14:10:35.495700315 (262h41m35.36678335s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.495951893 (262h41m35.367034929s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 16 14:10:35.495960940 (262h41m35.367043975s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.496269408 (262h41m35.367352443s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: context canceled; context canceled Dec 16 14:10:35.496278666 (262h41m35.367361701s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.496319834 (262h41m35.36740287s) [seg_pos={0 0}, upload=230290]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:10:35.496422099 (262h41m35.367505144s) [upload=230290]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:10:35.496427990 (262h41m35.367511035s) [upload=230290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:10:35.815631686 (262h41m35.686714751s) [upload=230290]: Done uploading segments... Dec 16 14:10:35.815655911 (262h41m35.686738966s) [upload=230290]: Upload finished. err: <nil> Duration: 6.288366623s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 42.575515458s trace: https://jaeger.dev.storj.io/trace/ba2c44dde12044 FAIL (2025-12-16 14:11:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=229725]: Uploading piece... Dec 16 13:30:17.600801266 (262h1m17.471884321s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.600906235 (262h1m17.471989281s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601269747 (262h1m17.472352782s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601276790 (262h1m17.472359835s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601630013 (262h1m17.472713048s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601640714 (262h1m17.472723749s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601965091 (262h1m17.473048126s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602047147 (262h1m17.473130192s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602246808 (262h1m17.473329853s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602601232 (262h1m17.473684277s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602716132 (262h1m17.473799167s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602996997 (262h1m17.474080042s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603027615 (262h1m17.47411065s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603255378 (262h1m17.474338413s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603501387 (262h1m17.474584432s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603959158 (262h1m17.475042203s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.604485590 (262h1m17.475568635s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:19.111089229 (262h1m18.982172264s) [upload=229725]: Next returned nil segment Dec 16 13:30:19.111104708 (262h1m18.982187753s) [upload=229725]: Waiting for error group managing segments... Dec 16 13:30:19.527483020 (262h1m19.398566075s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.621753699 (262h1m19.492836744s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.670564204 (262h1m19.541647249s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.672302895 (262h1m19.54338594s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.716239789 (262h1m19.587322834s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.735358003 (262h1m19.606441048s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.750731367 (262h1m19.621814412s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.764604575 (262h1m19.63568761s) [noise=true , address=45.91.116.7:4029 , piece_id=NZ5HRBRJ3QMNZMXJLKD5NPC3ZZFAS7VIDQNXPURRPWOCMPTXFUNQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.796960486 (262h1m19.668043531s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.812007538 (262h1m19.683090573s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.032271213 (262h1m19.903354248s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.099742841 (262h1m19.970825886s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.231931353 (262h1m20.103014398s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394678143 (262h1m20.265761188s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394706487 (262h1m20.265789532s) [seg_pos={0 0}, upload=229725]: Segment reached optimal threshold of 14 pieces. Dec 16 13:30:20.394971411 (262h1m20.266054466s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.394981671 (262h1m20.266064716s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395367956 (262h1m20.266451001s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395378747 (262h1m20.266461792s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395542348 (262h1m20.266625393s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395565903 (262h1m20.266648948s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395688968 (262h1m20.266772013s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395695741 (262h1m20.266778786s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395718144 (262h1m20.266801189s) [seg_pos={0 0}, upload=229725]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 13:30:20.395862569 (262h1m20.266945604s) [upload=229725]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:30:20.395896854 (262h1m20.266979899s) [upload=229725]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:30:20.679088774 (262h1m20.550171819s) [upload=229725]: Done uploading segments... Dec 16 13:30:20.679109594 (262h1m20.550192629s) [upload=229725]: Upload finished. err: <nil> Duration: 42.575530376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXX.X.X.XXX.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXXX] duration: 5.001497179s trace: https://jaeger.dev.storj.io/trace/4bd4daad74e9900c FAIL (2025-12-16 14:11:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001518199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..............................................X....X......X] duration: 30.004032842s trace: https://jaeger.dev.storj.io/trace/e49c6365710000b FAIL (2025-12-16 14:11:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ) context deadline exceeded ======LOGS====== Dec 16 14:10:32.985073926 (262h41m32.856156961s) [upload=230290]: Starting upload Dec 16 14:10:32.985154330 (262h41m32.856237375s) [upload=230290]: Done starting upload Dec 16 14:10:32.985233330 (262h41m32.856316365s) [upload=230290]: Uploading segments... Dec 16 14:10:32.985485239 (262h41m32.856568294s) [upload=230290]: Got next segment. Inline: false Dec 16 14:10:32.985601450 (262h41m32.856684485s) [upload=230290]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 14:10:32.985608184 (262h41m32.856691229s) [upload=230290]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 14:10:33.255392794 (262h41m33.126475829s) [seg_pos={0 0}, upload=230290]: Begin upload segment... Dec 16 14:10:33.255499337 (262h41m33.126582372s) [seg_pos={0 0}, upload=230290]: Done begin upload segment. Dec 16 14:10:33.255557117 (262h41m33.126640152s) [noise=true , address=145.79.227.51:4410 , piece_id=56VWGQ2SOLXWSNIXJMRI2VQ7VXYILL5EZN45IPSK6ZCWEJA4U2RA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.255585682 (262h41m33.126668727s) [noise=true , address=145.79.227.23:4545 , piece_id=PNRKCEAPMKBQTQMP7OOTS3S5MEW74VFREWCHZ5XAC7IDWF6S5MWA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.256346882 (262h41m33.127429917s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.256562532 (262h41m33.127645577s) [noise=true , address=145.79.227.40:4050 , piece_id=QPPTYML43XJRZCVQ5QKVTJMWOH4ETB6P2QUYQTPERBJ73OROLTWA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.257542648 (262h41m33.128625693s) [noise=true , address=45.91.116.6:4976 , piece_id=GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.258212524 (262h41m33.12929556s) [noise=true , address=145.79.227.17:4972 , piece_id=SS2ZYEXA2OWPY4MXJIT53HXHT5BZ5FBNFNECV3JVIPRTZCWRLXHQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.258731993 (262h41m33.129815028s) [noise=true , address=145.79.227.9:4365 , piece_id=7CHLK56ZSQ3VMLCHTGWEPXTYSOI7L6IX5IK7UUTFPYK434Y2BJ5Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259079935 (262h41m33.13016297s) [noise=true , address=145.79.227.45:4231 , piece_id=GI24XZVNFRHGDWBWFE3KE7W5OHBVEHLJ3B2BMCP6PNKGWJRAAYMA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259188884 (262h41m33.130271919s) [noise=true , address=45.91.116.14:4192 , piece_id=WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259521847 (262h41m33.130604892s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.259824013 (262h41m33.130907048s) [noise=true , address=145.79.227.38:4979 , piece_id=4NOR7PSPCHWIKHA5GGGPJA3FXNCNEXKWKOLBMORBEZG4NKXRTR4Q, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260168439 (262h41m33.131251484s) [noise=true , address=145.79.227.32:4836 , piece_id=F6L5YJ773REOLNMASD2JD7BTGFKB2UBXZMBPYERZ5I6DTYY74OMA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260470092 (262h41m33.131553137s) [noise=true , address=145.79.227.15:4876 , piece_id=DACMMDUBQG2TF2VERLEJ5CJXEWKGYTYTQ376KGICJNMCFFG4RWEA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.260751117 (262h41m33.131834163s) [noise=true , address=145.79.227.50:4374 , piece_id=O2T3JYB2C6RRAB3NXMQHBXF4EEGQES2KYPBGR6JM6SIIGFYNWPZQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.261047001 (262h41m33.132130036s) [noise=true , address=145.79.227.47:4267 , piece_id=CRCIH27Q7AYOJED5Q5Y3LB4GXOWJNEVAMXOGBTHEF6WKBGVW2EHQ, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:33.261378823 (262h41m33.132461858s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Uploading piece... Dec 16 14:10:34.973936146 (262h41m34.845019181s) [upload=230290]: Next returned nil segment Dec 16 14:10:34.973950845 (262h41m34.84503388s) [upload=230290]: Waiting for error group managing segments... Dec 16 14:10:35.255716630 (262h41m35.126799675s) [noise=true , address=145.79.227.45:4231 , piece_id=GI24XZVNFRHGDWBWFE3KE7W5OHBVEHLJ3B2BMCP6PNKGWJRAAYMA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.260286111 (262h41m35.131369156s) [noise=true , address=145.79.227.47:4267 , piece_id=CRCIH27Q7AYOJED5Q5Y3LB4GXOWJNEVAMXOGBTHEF6WKBGVW2EHQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.266206785 (262h41m35.13728982s) [noise=true , address=145.79.227.17:4972 , piece_id=SS2ZYEXA2OWPY4MXJIT53HXHT5BZ5FBNFNECV3JVIPRTZCWRLXHQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.269766433 (262h41m35.140849468s) [noise=true , address=145.79.227.15:4876 , piece_id=DACMMDUBQG2TF2VERLEJ5CJXEWKGYTYTQ376KGICJNMCFFG4RWEA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.277099697 (262h41m35.148182742s) [noise=true , address=145.79.227.50:4374 , piece_id=O2T3JYB2C6RRAB3NXMQHBXF4EEGQES2KYPBGR6JM6SIIGFYNWPZQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.277901374 (262h41m35.148984409s) [noise=true , address=145.79.227.32:4836 , piece_id=F6L5YJ773REOLNMASD2JD7BTGFKB2UBXZMBPYERZ5I6DTYY74OMA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.335067849 (262h41m35.206150884s) [noise=true , address=145.79.227.9:4365 , piece_id=7CHLK56ZSQ3VMLCHTGWEPXTYSOI7L6IX5IK7UUTFPYK434Y2BJ5Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.355162373 (262h41m35.226245408s) [noise=true , address=145.79.227.40:4050 , piece_id=QPPTYML43XJRZCVQ5QKVTJMWOH4ETB6P2QUYQTPERBJ73OROLTWA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.355598334 (262h41m35.226681379s) [noise=true , address=145.79.227.51:4410 , piece_id=56VWGQ2SOLXWSNIXJMRI2VQ7VXYILL5EZN45IPSK6ZCWEJA4U2RA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.381079334 (262h41m35.252162369s) [noise=true , address=145.79.227.23:4545 , piece_id=PNRKCEAPMKBQTQMP7OOTS3S5MEW74VFREWCHZ5XAC7IDWF6S5MWA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.389735517 (262h41m35.260818552s) [noise=true , address=145.79.227.38:4979 , piece_id=4NOR7PSPCHWIKHA5GGGPJA3FXNCNEXKWKOLBMORBEZG4NKXRTR4Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.476608340 (262h41m35.347691385s) [noise=true , address=45.91.116.14:4192 , piece_id=WARGVBFSSZ4CXIUHF4LQBUETX2524EPZSCCYE4CDFZJNZ4SILZGQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.495258663 (262h41m35.366341718s) [noise=true , address=45.91.116.6:4976 , piece_id=GTKWSOYQNDB3TVHNNYGCWDNANI6ALTQSXMMHG5OZCPAO6KEEW2NQ, seg_pos={0 0}, upload=230290]: Done uploading piece. err: <nil> Dec 16 14:10:35.495283441 (262h41m35.366366486s) [seg_pos={0 0}, upload=230290]: Segment reached optimal threshold of 13 pieces. Dec 16 14:10:35.495690136 (262h41m35.366773191s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 16 14:10:35.495700315 (262h41m35.36678335s) [noise=true , address=45.91.116.8:4046 , piece_id=2PNUOELYOVSDUT3HHUIL2OEYLUJVFOK2VBKGGWCAHICOVMOV5N3Q, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.495951893 (262h41m35.367034929s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 16 14:10:35.495960940 (262h41m35.367043975s) [noise=true , address=45.91.116.9:4262 , piece_id=FPREDFJQFZ6YREJXEC4VLKHKMVABQFFCNYUFPZHJJZ227V44JR6Q, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.496269408 (262h41m35.367352443s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: context canceled; context canceled Dec 16 14:10:35.496278666 (262h41m35.367361701s) [noise=true , address=45.91.116.12:4228 , piece_id=Y6UH2NHWPCFEMQRNWYANNQKPAFBWLS26CHAQOHWHTDR5FTIKT7WA, seg_pos={0 0}, upload=230290]: Long tail cancellation due to optimal threshold Dec 16 14:10:35.496319834 (262h41m35.36740287s) [seg_pos={0 0}, upload=230290]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 14:10:35.496422099 (262h41m35.367505144s) [upload=230290]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 14:10:35.496427990 (262h41m35.367511035s) [upload=230290]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 14:10:35.815631686 (262h41m35.686714751s) [upload=230290]: Done uploading segments... Dec 16 14:10:35.815655911 (262h41m35.686738966s) [upload=230290]: Upload finished. err: <nil> Duration: 30.004049253s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................X.XX....X.............] duration: 14.712265795s trace: https://jaeger.dev.storj.io/trace/24f03ce3e3f5cb2d FAIL (2025-12-16 14:11:01 UTC) orbiter warning: context deadline exceeded Duration: 14.712296473s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXX.X.X.XXX.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XXX] duration: 5.001516385s trace: https://jaeger.dev.storj.io/trace/195c04a6aa854fb1 FAIL (2025-12-16 14:10:50 UTC) orbiter warning: context deadline exceeded Duration: 5.001543728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 45.12073077s trace: https://jaeger.dev.storj.io/trace/ba2c44dde12044 FAIL (2025-12-16 14:10:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=229725]: Uploading piece... Dec 16 13:30:17.600801266 (262h1m17.471884321s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.600906235 (262h1m17.471989281s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601269747 (262h1m17.472352782s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601276790 (262h1m17.472359835s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601630013 (262h1m17.472713048s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601640714 (262h1m17.472723749s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.601965091 (262h1m17.473048126s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602047147 (262h1m17.473130192s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602246808 (262h1m17.473329853s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602601232 (262h1m17.473684277s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602716132 (262h1m17.473799167s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.602996997 (262h1m17.474080042s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603027615 (262h1m17.47411065s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603255378 (262h1m17.474338413s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603501387 (262h1m17.474584432s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.603959158 (262h1m17.475042203s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:17.604485590 (262h1m17.475568635s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Uploading piece... Dec 16 13:30:19.111089229 (262h1m18.982172264s) [upload=229725]: Next returned nil segment Dec 16 13:30:19.111104708 (262h1m18.982187753s) [upload=229725]: Waiting for error group managing segments... Dec 16 13:30:19.527483020 (262h1m19.398566075s) [noise=true , address=5.231.73.5:4026 , piece_id=ZUAKOCC4QU3ZNG3FDRQXSXXCPUBR2NY5GR3ZRWH5HREQPNNHNFFA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.621753699 (262h1m19.492836744s) [noise=true , address=145.79.227.33:4887 , piece_id=CRZ6DQFLOHSAQXJQYRRWEMZQY7SJKJ55URX47SD6C7GYHYMQ6Z5Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.670564204 (262h1m19.541647249s) [noise=true , address=77.90.31.16:4768 , piece_id=7M3WEIJODFX3ZHOPCHAY5ZMB4KOEJA3L4RZMM4XFGLSVGDE4NUQQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.672302895 (262h1m19.54338594s) [noise=true , address=5.231.73.4:4007 , piece_id=LQ3LUUW5L2AXQXZDEQDPSHTIIHYBUQXGFIU2WPRJZTUKL4TEVHGQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.716239789 (262h1m19.587322834s) [noise=true , address=77.90.31.11:4570 , piece_id=RMNIKRVXJ5S6D2IJEELGJOTZITRBXMF7RSLI77WIDEFZURDVM3EQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.735358003 (262h1m19.606441048s) [noise=true , address=45.91.116.5:4308 , piece_id=NJK4HJASXD7TIKYAX3HW2XHWV4VUEZZGKSPROXXHQMOYLL5TBD5A, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.750731367 (262h1m19.621814412s) [noise=true , address=77.90.31.12:4621 , piece_id=LQO4J7OHLI3SU5IHQHHVEQBQRA43XTKS6KKMAGBHYGC77GAX3C4Q, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.764604575 (262h1m19.63568761s) [noise=true , address=45.91.116.7:4029 , piece_id=NZ5HRBRJ3QMNZMXJLKD5NPC3ZZFAS7VIDQNXPURRPWOCMPTXFUNQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.796960486 (262h1m19.668043531s) [noise=true , address=77.90.31.13:4644 , piece_id=2QRBQDMBU7DDBOPTEGSXWHLQ6EHJYFOPSVKQZILRDGSIG2R6U4SQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:19.812007538 (262h1m19.683090573s) [noise=true , address=45.91.116.11:4121 , piece_id=SUM6WGGYDG365Q47QN6DYCYS2FLT3R26PLMYAMDNZET53UYNIKVQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.032271213 (262h1m19.903354248s) [noise=true , address=64.49.13.26:4952 , piece_id=YUKWL5ULV7DJE3F3PMMCV7XJCJKCEZVHRVXTHSKE72OBOMLOTBSQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.099742841 (262h1m19.970825886s) [noise=true , address=45.91.116.6:4993 , piece_id=NXWPRYVOO7DATJ67ZOI64NSK4EG5XY432Z6FKDQKMRG3VULUZYTQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.231931353 (262h1m20.103014398s) [noise=true , address=77.90.31.10:4526 , piece_id=MW354UQUAPW764MN3QBFAESGQBBC4EBJUES4G4ZTXM6YJLBEUWTA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394678143 (262h1m20.265761188s) [noise=true , address=145.79.227.30:4777 , piece_id=REKQ2D2SZI7HBWAMWY2HRRUMYTCGIP4J4ABWQJQN2MN43YSN66YQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: <nil> Dec 16 13:30:20.394706487 (262h1m20.265789532s) [seg_pos={0 0}, upload=229725]: Segment reached optimal threshold of 14 pieces. Dec 16 13:30:20.394971411 (262h1m20.266054466s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F8hBt2nRQBHGzba5pPKyYJF37b7gRUnmBoZfDAYZAkYaUfk2V): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.394981671 (262h1m20.266064716s) [noise=true , address=77.90.31.19:4849 , piece_id=7EI7KU4CAJBBL2Z4DAFSNTUXK5H76CYL3RNRMRSJC3H2XQBTAUDQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395367956 (262h1m20.266451001s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395378747 (262h1m20.266461792s) [noise=true , address=77.90.31.21:4882 , piece_id=ZRYYBZ4WOSSKCGHES6VUFGNEY7OTZ633VI4R4BDQKLYJMGJLL3KA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395542348 (262h1m20.266625393s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bquxzprFQQwaeW4KbSR5i8CBP6XCcHAWPmAmoszcsLUWKKT5g): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395565903 (262h1m20.266648948s) [noise=true , address=45.91.116.9:4283 , piece_id=UPUKGH53IVPBTRDGG6DASHHGI6LUC5G2UIOUYUE7F7742I3ZQ5IA, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395688968 (262h1m20.266772013s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Dec 16 13:30:20.395695741 (262h1m20.266778786s) [noise=true , address=77.90.31.18:4815 , piece_id=7IT5PZ2PXR7KJLPKSLBJLCNACJBUCD4V3MHR2ELEYCSLNBQL2XPQ, seg_pos={0 0}, upload=229725]: Long tail cancellation due to optimal threshold Dec 16 13:30:20.395718144 (262h1m20.266801189s) [seg_pos={0 0}, upload=229725]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 16 13:30:20.395862569 (262h1m20.266945604s) [upload=229725]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 13:30:20.395896854 (262h1m20.266979899s) [upload=229725]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 13:30:20.679088774 (262h1m20.550171819s) [upload=229725]: Done uploading segments... Dec 16 13:30:20.679109594 (262h1m20.550192629s) [upload=229725]: Upload finished. err: <nil> Duration: 45.120756078s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXXX.X.X.XXX.XXXX....XXXXX.XX..X..XXX.XXXXXX..XX.XXX.XX.XX] duration: 5.001196718s trace: https://jaeger.dev.storj.io/trace/2782e3ac3a34b8db FAIL (2025-12-16 14:10:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001215794s core uplink us-select-1 us1-placement-20

1
...
11489
11490
11491
11492
11493
...
17599