Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X....X...XX..X..XX...X....................X..............] duration: 5.000989488s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:26:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00109987s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....X....X...XX..X..XX...X....................X.............] duration: 3.789089437s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:26:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.789115217s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X......................XX..........XX.........] duration: 21.889534141s trace: https://jaeger.dev.storj.io/trace/379227ab2c9a3c9c FAIL (2025-12-17 17:26:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17 17:18:37.302512283 (289h49m37.173595329s) [upload=254087]: Starting upload Dec 17 17:18:37.302642772 (289h49m37.173725817s) [upload=254087]: Done starting upload Dec 17 17:18:37.302700041 (289h49m37.173783086s) [upload=254087]: Uploading segments... Dec 17 17:18:37.302889491 (289h49m37.173972537s) [upload=254087]: Got next segment. Inline: false Dec 17 17:18:37.302931863 (289h49m37.174014918s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:18:37.302937634 (289h49m37.174020669s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:18:37.569724064 (289h49m37.440807099s) [seg_pos={0 0}, upload=254087]: Begin upload segment... Dec 17 17:18:37.569770663 (289h49m37.440853708s) [seg_pos={0 0}, upload=254087]: Done begin upload segment. Dec 17 17:18:37.569840656 (289h49m37.440923691s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569858651 (289h49m37.440941696s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569957709 (289h49m37.441040754s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570381687 (289h49m37.441464732s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570721273 (289h49m37.441804319s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571012067 (289h49m37.442095102s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571295546 (289h49m37.442378592s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571934833 (289h49m37.443017878s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572398788 (289h49m37.443481833s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572844126 (289h49m37.443927171s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.573930164 (289h49m37.445013209s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.574335204 (289h49m37.445418249s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575519129 (289h49m37.446602174s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575802970 (289h49m37.446886015s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.578052403 (289h49m37.449135448s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.579787497 (289h49m37.450870542s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:39.276574183 (289h49m39.147657218s) [upload=254087]: Next returned nil segment Dec 17 17:18:39.276593710 (289h49m39.147676765s) [upload=254087]: Waiting for error group managing segments... Dec 17 17:18:39.583499194 (289h49m39.45458224s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.586064850 (289h49m39.457147885s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.627263460 (289h49m39.498346505s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.653542299 (289h49m39.524625355s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.668227403 (289h49m39.539310448s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.776965220 (289h49m39.648048256s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.827312301 (289h49m39.698395336s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.828976920 (289h49m39.700059955s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841154408 (289h49m39.712237453s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841375590 (289h49m39.712458625s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.844226457 (289h49m39.715309492s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.865188934 (289h49m39.736271979s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882207611 (289h49m39.753290656s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882235925 (289h49m39.75331896s) [seg_pos={0 0}, upload=254087]: Segment reached optimal threshold of 13 pieces. Dec 17 17:18:39.882482044 (289h49m39.753565099s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection Dec 17 17:18:39.882491832 (289h49m39.753574877s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882765573 (289h49m39.753848618s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882804698 (289h49m39.753887743s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882826319 (289h49m39.753909485s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882837600 (289h49m39.753920645s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882883669 (289h49m39.753966714s) [seg_pos={0 0}, upload=254087]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:18:39.883006001 (289h49m39.754089056s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:18:39.883050125 (289h49m39.75413317s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:18:40.198144112 (289h49m40.069227157s) [upload=254087]: Done uploading segments... Dec 17 17:18:40.198162958 (289h49m40.069245993s) [upload=254087]: Upload finished. err: <nil> Duration: 21.889838039s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 7.826189039s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:25:43 UTC) orbiter warning: context deadline exceeded Duration: 7.826217242s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 7.247776796s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:25:18 UTC) orbiter warning: context deadline exceeded Duration: 7.247802305s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....X....X...XX..X..XX...X....................X............] duration: 3.506479103s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:25:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.506507498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X......................XX..........XX........] duration: 7.627741492s trace: https://jaeger.dev.storj.io/trace/379227ab2c9a3c9c FAIL (2025-12-17 17:24:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17 17:18:37.302512283 (289h49m37.173595329s) [upload=254087]: Starting upload Dec 17 17:18:37.302642772 (289h49m37.173725817s) [upload=254087]: Done starting upload Dec 17 17:18:37.302700041 (289h49m37.173783086s) [upload=254087]: Uploading segments... Dec 17 17:18:37.302889491 (289h49m37.173972537s) [upload=254087]: Got next segment. Inline: false Dec 17 17:18:37.302931863 (289h49m37.174014918s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:18:37.302937634 (289h49m37.174020669s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:18:37.569724064 (289h49m37.440807099s) [seg_pos={0 0}, upload=254087]: Begin upload segment... Dec 17 17:18:37.569770663 (289h49m37.440853708s) [seg_pos={0 0}, upload=254087]: Done begin upload segment. Dec 17 17:18:37.569840656 (289h49m37.440923691s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569858651 (289h49m37.440941696s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569957709 (289h49m37.441040754s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570381687 (289h49m37.441464732s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570721273 (289h49m37.441804319s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571012067 (289h49m37.442095102s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571295546 (289h49m37.442378592s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571934833 (289h49m37.443017878s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572398788 (289h49m37.443481833s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572844126 (289h49m37.443927171s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.573930164 (289h49m37.445013209s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.574335204 (289h49m37.445418249s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575519129 (289h49m37.446602174s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575802970 (289h49m37.446886015s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.578052403 (289h49m37.449135448s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.579787497 (289h49m37.450870542s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:39.276574183 (289h49m39.147657218s) [upload=254087]: Next returned nil segment Dec 17 17:18:39.276593710 (289h49m39.147676765s) [upload=254087]: Waiting for error group managing segments... Dec 17 17:18:39.583499194 (289h49m39.45458224s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.586064850 (289h49m39.457147885s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.627263460 (289h49m39.498346505s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.653542299 (289h49m39.524625355s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.668227403 (289h49m39.539310448s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.776965220 (289h49m39.648048256s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.827312301 (289h49m39.698395336s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.828976920 (289h49m39.700059955s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841154408 (289h49m39.712237453s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841375590 (289h49m39.712458625s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.844226457 (289h49m39.715309492s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.865188934 (289h49m39.736271979s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882207611 (289h49m39.753290656s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882235925 (289h49m39.75331896s) [seg_pos={0 0}, upload=254087]: Segment reached optimal threshold of 13 pieces. Dec 17 17:18:39.882482044 (289h49m39.753565099s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection Dec 17 17:18:39.882491832 (289h49m39.753574877s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882765573 (289h49m39.753848618s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882804698 (289h49m39.753887743s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882826319 (289h49m39.753909485s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882837600 (289h49m39.753920645s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882883669 (289h49m39.753966714s) [seg_pos={0 0}, upload=254087]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:18:39.883006001 (289h49m39.754089056s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:18:39.883050125 (289h49m39.75413317s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:18:40.198144112 (289h49m40.069227157s) [upload=254087]: Done uploading segments... Dec 17 17:18:40.198162958 (289h49m40.069245993s) [upload=254087]: Upload finished. err: <nil> Duration: 7.627758643s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X.....X....X...XX..X..XX...X....................X...........] duration: 3.608146869s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:24:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.608169383s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.948765767s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:24:45 UTC) orbiter warning: context deadline exceeded Duration: 6.948790724s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [XX.....X....X...XX..X..XX...X....................X..........] duration: 3.488977619s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:24:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.489037113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X......................XX..........XX.......] duration: 5.827858878s trace: https://jaeger.dev.storj.io/trace/379227ab2c9a3c9c FAIL (2025-12-17 17:24:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17 17:18:37.302512283 (289h49m37.173595329s) [upload=254087]: Starting upload Dec 17 17:18:37.302642772 (289h49m37.173725817s) [upload=254087]: Done starting upload Dec 17 17:18:37.302700041 (289h49m37.173783086s) [upload=254087]: Uploading segments... Dec 17 17:18:37.302889491 (289h49m37.173972537s) [upload=254087]: Got next segment. Inline: false Dec 17 17:18:37.302931863 (289h49m37.174014918s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:18:37.302937634 (289h49m37.174020669s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:18:37.569724064 (289h49m37.440807099s) [seg_pos={0 0}, upload=254087]: Begin upload segment... Dec 17 17:18:37.569770663 (289h49m37.440853708s) [seg_pos={0 0}, upload=254087]: Done begin upload segment. Dec 17 17:18:37.569840656 (289h49m37.440923691s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569858651 (289h49m37.440941696s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569957709 (289h49m37.441040754s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570381687 (289h49m37.441464732s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570721273 (289h49m37.441804319s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571012067 (289h49m37.442095102s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571295546 (289h49m37.442378592s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571934833 (289h49m37.443017878s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572398788 (289h49m37.443481833s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572844126 (289h49m37.443927171s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.573930164 (289h49m37.445013209s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.574335204 (289h49m37.445418249s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575519129 (289h49m37.446602174s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575802970 (289h49m37.446886015s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.578052403 (289h49m37.449135448s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.579787497 (289h49m37.450870542s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:39.276574183 (289h49m39.147657218s) [upload=254087]: Next returned nil segment Dec 17 17:18:39.276593710 (289h49m39.147676765s) [upload=254087]: Waiting for error group managing segments... Dec 17 17:18:39.583499194 (289h49m39.45458224s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.586064850 (289h49m39.457147885s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.627263460 (289h49m39.498346505s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.653542299 (289h49m39.524625355s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.668227403 (289h49m39.539310448s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.776965220 (289h49m39.648048256s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.827312301 (289h49m39.698395336s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.828976920 (289h49m39.700059955s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841154408 (289h49m39.712237453s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841375590 (289h49m39.712458625s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.844226457 (289h49m39.715309492s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.865188934 (289h49m39.736271979s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882207611 (289h49m39.753290656s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882235925 (289h49m39.75331896s) [seg_pos={0 0}, upload=254087]: Segment reached optimal threshold of 13 pieces. Dec 17 17:18:39.882482044 (289h49m39.753565099s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection Dec 17 17:18:39.882491832 (289h49m39.753574877s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882765573 (289h49m39.753848618s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882804698 (289h49m39.753887743s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882826319 (289h49m39.753909485s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882837600 (289h49m39.753920645s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882883669 (289h49m39.753966714s) [seg_pos={0 0}, upload=254087]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:18:39.883006001 (289h49m39.754089056s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:18:39.883050125 (289h49m39.75413317s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:18:40.198144112 (289h49m40.069227157s) [upload=254087]: Done uploading segments... Dec 17 17:18:40.198162958 (289h49m40.069245993s) [upload=254087]: Upload finished. err: <nil> Duration: 5.82788549s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.161112197s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:23:47 UTC) orbiter warning: context deadline exceeded Duration: 6.161138646s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 9.860295322s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:23:33 UTC) orbiter warning: context deadline exceeded Duration: 9.86033132s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.XX.....X....X...XX..X..XX...X....................X.........] duration: 3.616500568s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:23:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.616542648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X......................XX..........XX......] duration: 5.980530105s trace: https://jaeger.dev.storj.io/trace/379227ab2c9a3c9c FAIL (2025-12-17 17:23:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17 17:18:37.302512283 (289h49m37.173595329s) [upload=254087]: Starting upload Dec 17 17:18:37.302642772 (289h49m37.173725817s) [upload=254087]: Done starting upload Dec 17 17:18:37.302700041 (289h49m37.173783086s) [upload=254087]: Uploading segments... Dec 17 17:18:37.302889491 (289h49m37.173972537s) [upload=254087]: Got next segment. Inline: false Dec 17 17:18:37.302931863 (289h49m37.174014918s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:18:37.302937634 (289h49m37.174020669s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:18:37.569724064 (289h49m37.440807099s) [seg_pos={0 0}, upload=254087]: Begin upload segment... Dec 17 17:18:37.569770663 (289h49m37.440853708s) [seg_pos={0 0}, upload=254087]: Done begin upload segment. Dec 17 17:18:37.569840656 (289h49m37.440923691s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569858651 (289h49m37.440941696s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569957709 (289h49m37.441040754s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570381687 (289h49m37.441464732s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570721273 (289h49m37.441804319s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571012067 (289h49m37.442095102s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571295546 (289h49m37.442378592s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571934833 (289h49m37.443017878s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572398788 (289h49m37.443481833s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572844126 (289h49m37.443927171s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.573930164 (289h49m37.445013209s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.574335204 (289h49m37.445418249s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575519129 (289h49m37.446602174s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575802970 (289h49m37.446886015s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.578052403 (289h49m37.449135448s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.579787497 (289h49m37.450870542s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:39.276574183 (289h49m39.147657218s) [upload=254087]: Next returned nil segment Dec 17 17:18:39.276593710 (289h49m39.147676765s) [upload=254087]: Waiting for error group managing segments... Dec 17 17:18:39.583499194 (289h49m39.45458224s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.586064850 (289h49m39.457147885s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.627263460 (289h49m39.498346505s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.653542299 (289h49m39.524625355s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.668227403 (289h49m39.539310448s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.776965220 (289h49m39.648048256s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.827312301 (289h49m39.698395336s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.828976920 (289h49m39.700059955s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841154408 (289h49m39.712237453s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841375590 (289h49m39.712458625s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.844226457 (289h49m39.715309492s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.865188934 (289h49m39.736271979s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882207611 (289h49m39.753290656s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882235925 (289h49m39.75331896s) [seg_pos={0 0}, upload=254087]: Segment reached optimal threshold of 13 pieces. Dec 17 17:18:39.882482044 (289h49m39.753565099s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection Dec 17 17:18:39.882491832 (289h49m39.753574877s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882765573 (289h49m39.753848618s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882804698 (289h49m39.753887743s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882826319 (289h49m39.753909485s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882837600 (289h49m39.753920645s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882883669 (289h49m39.753966714s) [seg_pos={0 0}, upload=254087]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:18:39.883006001 (289h49m39.754089056s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:18:39.883050125 (289h49m39.75413317s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:18:40.198144112 (289h49m40.069227157s) [upload=254087]: Done uploading segments... Dec 17 17:18:40.198162958 (289h49m40.069245993s) [upload=254087]: Upload finished. err: <nil> Duration: 5.980558369s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 11.775183296s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:23:11 UTC) orbiter warning: context deadline exceeded Duration: 11.775209386s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X......................XX..........XX.....] duration: 5.969353083s trace: https://jaeger.dev.storj.io/trace/379227ab2c9a3c9c FAIL (2025-12-17 17:22:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17 17:18:37.302512283 (289h49m37.173595329s) [upload=254087]: Starting upload Dec 17 17:18:37.302642772 (289h49m37.173725817s) [upload=254087]: Done starting upload Dec 17 17:18:37.302700041 (289h49m37.173783086s) [upload=254087]: Uploading segments... Dec 17 17:18:37.302889491 (289h49m37.173972537s) [upload=254087]: Got next segment. Inline: false Dec 17 17:18:37.302931863 (289h49m37.174014918s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:18:37.302937634 (289h49m37.174020669s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:18:37.569724064 (289h49m37.440807099s) [seg_pos={0 0}, upload=254087]: Begin upload segment... Dec 17 17:18:37.569770663 (289h49m37.440853708s) [seg_pos={0 0}, upload=254087]: Done begin upload segment. Dec 17 17:18:37.569840656 (289h49m37.440923691s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569858651 (289h49m37.440941696s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569957709 (289h49m37.441040754s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570381687 (289h49m37.441464732s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570721273 (289h49m37.441804319s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571012067 (289h49m37.442095102s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571295546 (289h49m37.442378592s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571934833 (289h49m37.443017878s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572398788 (289h49m37.443481833s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572844126 (289h49m37.443927171s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.573930164 (289h49m37.445013209s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.574335204 (289h49m37.445418249s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575519129 (289h49m37.446602174s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575802970 (289h49m37.446886015s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.578052403 (289h49m37.449135448s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.579787497 (289h49m37.450870542s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:39.276574183 (289h49m39.147657218s) [upload=254087]: Next returned nil segment Dec 17 17:18:39.276593710 (289h49m39.147676765s) [upload=254087]: Waiting for error group managing segments... Dec 17 17:18:39.583499194 (289h49m39.45458224s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.586064850 (289h49m39.457147885s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.627263460 (289h49m39.498346505s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.653542299 (289h49m39.524625355s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.668227403 (289h49m39.539310448s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.776965220 (289h49m39.648048256s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.827312301 (289h49m39.698395336s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.828976920 (289h49m39.700059955s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841154408 (289h49m39.712237453s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841375590 (289h49m39.712458625s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.844226457 (289h49m39.715309492s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.865188934 (289h49m39.736271979s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882207611 (289h49m39.753290656s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882235925 (289h49m39.75331896s) [seg_pos={0 0}, upload=254087]: Segment reached optimal threshold of 13 pieces. Dec 17 17:18:39.882482044 (289h49m39.753565099s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection Dec 17 17:18:39.882491832 (289h49m39.753574877s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882765573 (289h49m39.753848618s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882804698 (289h49m39.753887743s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882826319 (289h49m39.753909485s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882837600 (289h49m39.753920645s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882883669 (289h49m39.753966714s) [seg_pos={0 0}, upload=254087]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:18:39.883006001 (289h49m39.754089056s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:18:39.883050125 (289h49m39.75413317s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:18:40.198144112 (289h49m40.069227157s) [upload=254087]: Done uploading segments... Dec 17 17:18:40.198162958 (289h49m40.069245993s) [upload=254087]: Upload finished. err: <nil> Duration: 5.969380496s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..XX.....X....X...XX..X..XX...X....................X........] duration: 3.633899479s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:22:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.633930577s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 10.256018739s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:22:09 UTC) orbiter warning: context deadline exceeded Duration: 10.256046812s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X......................XX..........XX....] duration: 5.893773066s trace: https://jaeger.dev.storj.io/trace/379227ab2c9a3c9c FAIL (2025-12-17 17:21:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17 17:18:37.302512283 (289h49m37.173595329s) [upload=254087]: Starting upload Dec 17 17:18:37.302642772 (289h49m37.173725817s) [upload=254087]: Done starting upload Dec 17 17:18:37.302700041 (289h49m37.173783086s) [upload=254087]: Uploading segments... Dec 17 17:18:37.302889491 (289h49m37.173972537s) [upload=254087]: Got next segment. Inline: false Dec 17 17:18:37.302931863 (289h49m37.174014918s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:18:37.302937634 (289h49m37.174020669s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:18:37.569724064 (289h49m37.440807099s) [seg_pos={0 0}, upload=254087]: Begin upload segment... Dec 17 17:18:37.569770663 (289h49m37.440853708s) [seg_pos={0 0}, upload=254087]: Done begin upload segment. Dec 17 17:18:37.569840656 (289h49m37.440923691s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569858651 (289h49m37.440941696s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569957709 (289h49m37.441040754s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570381687 (289h49m37.441464732s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570721273 (289h49m37.441804319s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571012067 (289h49m37.442095102s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571295546 (289h49m37.442378592s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571934833 (289h49m37.443017878s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572398788 (289h49m37.443481833s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572844126 (289h49m37.443927171s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.573930164 (289h49m37.445013209s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.574335204 (289h49m37.445418249s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575519129 (289h49m37.446602174s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575802970 (289h49m37.446886015s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.578052403 (289h49m37.449135448s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.579787497 (289h49m37.450870542s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:39.276574183 (289h49m39.147657218s) [upload=254087]: Next returned nil segment Dec 17 17:18:39.276593710 (289h49m39.147676765s) [upload=254087]: Waiting for error group managing segments... Dec 17 17:18:39.583499194 (289h49m39.45458224s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.586064850 (289h49m39.457147885s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.627263460 (289h49m39.498346505s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.653542299 (289h49m39.524625355s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.668227403 (289h49m39.539310448s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.776965220 (289h49m39.648048256s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.827312301 (289h49m39.698395336s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.828976920 (289h49m39.700059955s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841154408 (289h49m39.712237453s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841375590 (289h49m39.712458625s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.844226457 (289h49m39.715309492s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.865188934 (289h49m39.736271979s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882207611 (289h49m39.753290656s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882235925 (289h49m39.75331896s) [seg_pos={0 0}, upload=254087]: Segment reached optimal threshold of 13 pieces. Dec 17 17:18:39.882482044 (289h49m39.753565099s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection Dec 17 17:18:39.882491832 (289h49m39.753574877s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882765573 (289h49m39.753848618s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882804698 (289h49m39.753887743s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882826319 (289h49m39.753909485s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882837600 (289h49m39.753920645s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882883669 (289h49m39.753966714s) [seg_pos={0 0}, upload=254087]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:18:39.883006001 (289h49m39.754089056s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:18:39.883050125 (289h49m39.75413317s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:18:40.198144112 (289h49m40.069227157s) [upload=254087]: Done uploading segments... Dec 17 17:18:40.198162958 (289h49m40.069245993s) [upload=254087]: Upload finished. err: <nil> Duration: 5.893808042s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X..XX.....X....X...XX..X..XX...X....................X.......] duration: 3.674292232s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:21:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.674337278s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 8.63190099s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:21:21 UTC) orbiter warning: context deadline exceeded Duration: 8.631935826s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X......................XX..........XX...] duration: 5.811228463s trace: https://jaeger.dev.storj.io/trace/379227ab2c9a3c9c FAIL (2025-12-17 17:21:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17 17:18:37.302512283 (289h49m37.173595329s) [upload=254087]: Starting upload Dec 17 17:18:37.302642772 (289h49m37.173725817s) [upload=254087]: Done starting upload Dec 17 17:18:37.302700041 (289h49m37.173783086s) [upload=254087]: Uploading segments... Dec 17 17:18:37.302889491 (289h49m37.173972537s) [upload=254087]: Got next segment. Inline: false Dec 17 17:18:37.302931863 (289h49m37.174014918s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:18:37.302937634 (289h49m37.174020669s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:18:37.569724064 (289h49m37.440807099s) [seg_pos={0 0}, upload=254087]: Begin upload segment... Dec 17 17:18:37.569770663 (289h49m37.440853708s) [seg_pos={0 0}, upload=254087]: Done begin upload segment. Dec 17 17:18:37.569840656 (289h49m37.440923691s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569858651 (289h49m37.440941696s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569957709 (289h49m37.441040754s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570381687 (289h49m37.441464732s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570721273 (289h49m37.441804319s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571012067 (289h49m37.442095102s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571295546 (289h49m37.442378592s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571934833 (289h49m37.443017878s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572398788 (289h49m37.443481833s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572844126 (289h49m37.443927171s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.573930164 (289h49m37.445013209s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.574335204 (289h49m37.445418249s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575519129 (289h49m37.446602174s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575802970 (289h49m37.446886015s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.578052403 (289h49m37.449135448s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.579787497 (289h49m37.450870542s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:39.276574183 (289h49m39.147657218s) [upload=254087]: Next returned nil segment Dec 17 17:18:39.276593710 (289h49m39.147676765s) [upload=254087]: Waiting for error group managing segments... Dec 17 17:18:39.583499194 (289h49m39.45458224s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.586064850 (289h49m39.457147885s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.627263460 (289h49m39.498346505s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.653542299 (289h49m39.524625355s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.668227403 (289h49m39.539310448s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.776965220 (289h49m39.648048256s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.827312301 (289h49m39.698395336s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.828976920 (289h49m39.700059955s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841154408 (289h49m39.712237453s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841375590 (289h49m39.712458625s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.844226457 (289h49m39.715309492s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.865188934 (289h49m39.736271979s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882207611 (289h49m39.753290656s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882235925 (289h49m39.75331896s) [seg_pos={0 0}, upload=254087]: Segment reached optimal threshold of 13 pieces. Dec 17 17:18:39.882482044 (289h49m39.753565099s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection Dec 17 17:18:39.882491832 (289h49m39.753574877s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882765573 (289h49m39.753848618s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882804698 (289h49m39.753887743s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882826319 (289h49m39.753909485s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882837600 (289h49m39.753920645s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882883669 (289h49m39.753966714s) [seg_pos={0 0}, upload=254087]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:18:39.883006001 (289h49m39.754089056s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:18:39.883050125 (289h49m39.75413317s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:18:40.198144112 (289h49m40.069227157s) [upload=254087]: Done uploading segments... Dec 17 17:18:40.198162958 (289h49m40.069245993s) [upload=254087]: Upload finished. err: <nil> Duration: 5.811258801s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 12.079686688s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:20:42 UTC) orbiter warning: context deadline exceeded Duration: 12.079721905s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 12.115328127s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:20:21 UTC) orbiter warning: context deadline exceeded Duration: 12.115355599s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [XX..XX.....X....X...XX..X..XX...X....................X......] duration: 3.562892707s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:20:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.562923776s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................X......................XX..........XX..] duration: 5.902705734s trace: https://jaeger.dev.storj.io/trace/379227ab2c9a3c9c FAIL (2025-12-17 17:20:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17 17:18:37.302512283 (289h49m37.173595329s) [upload=254087]: Starting upload Dec 17 17:18:37.302642772 (289h49m37.173725817s) [upload=254087]: Done starting upload Dec 17 17:18:37.302700041 (289h49m37.173783086s) [upload=254087]: Uploading segments... Dec 17 17:18:37.302889491 (289h49m37.173972537s) [upload=254087]: Got next segment. Inline: false Dec 17 17:18:37.302931863 (289h49m37.174014918s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:18:37.302937634 (289h49m37.174020669s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:18:37.569724064 (289h49m37.440807099s) [seg_pos={0 0}, upload=254087]: Begin upload segment... Dec 17 17:18:37.569770663 (289h49m37.440853708s) [seg_pos={0 0}, upload=254087]: Done begin upload segment. Dec 17 17:18:37.569840656 (289h49m37.440923691s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569858651 (289h49m37.440941696s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569957709 (289h49m37.441040754s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570381687 (289h49m37.441464732s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570721273 (289h49m37.441804319s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571012067 (289h49m37.442095102s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571295546 (289h49m37.442378592s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571934833 (289h49m37.443017878s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572398788 (289h49m37.443481833s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572844126 (289h49m37.443927171s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.573930164 (289h49m37.445013209s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.574335204 (289h49m37.445418249s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575519129 (289h49m37.446602174s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575802970 (289h49m37.446886015s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.578052403 (289h49m37.449135448s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.579787497 (289h49m37.450870542s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:39.276574183 (289h49m39.147657218s) [upload=254087]: Next returned nil segment Dec 17 17:18:39.276593710 (289h49m39.147676765s) [upload=254087]: Waiting for error group managing segments... Dec 17 17:18:39.583499194 (289h49m39.45458224s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.586064850 (289h49m39.457147885s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.627263460 (289h49m39.498346505s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.653542299 (289h49m39.524625355s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.668227403 (289h49m39.539310448s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.776965220 (289h49m39.648048256s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.827312301 (289h49m39.698395336s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.828976920 (289h49m39.700059955s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841154408 (289h49m39.712237453s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841375590 (289h49m39.712458625s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.844226457 (289h49m39.715309492s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.865188934 (289h49m39.736271979s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882207611 (289h49m39.753290656s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882235925 (289h49m39.75331896s) [seg_pos={0 0}, upload=254087]: Segment reached optimal threshold of 13 pieces. Dec 17 17:18:39.882482044 (289h49m39.753565099s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection Dec 17 17:18:39.882491832 (289h49m39.753574877s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882765573 (289h49m39.753848618s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882804698 (289h49m39.753887743s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882826319 (289h49m39.753909485s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882837600 (289h49m39.753920645s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882883669 (289h49m39.753966714s) [seg_pos={0 0}, upload=254087]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:18:39.883006001 (289h49m39.754089056s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:18:39.883050125 (289h49m39.75413317s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:18:40.198144112 (289h49m40.069227157s) [upload=254087]: Done uploading segments... Dec 17 17:18:40.198162958 (289h49m40.069245993s) [upload=254087]: Upload finished. err: <nil> Duration: 5.902741022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................X......................XX..........XX.] duration: 18.300937054s trace: https://jaeger.dev.storj.io/trace/379227ab2c9a3c9c FAIL (2025-12-17 17:19:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17 17:18:37.302512283 (289h49m37.173595329s) [upload=254087]: Starting upload Dec 17 17:18:37.302642772 (289h49m37.173725817s) [upload=254087]: Done starting upload Dec 17 17:18:37.302700041 (289h49m37.173783086s) [upload=254087]: Uploading segments... Dec 17 17:18:37.302889491 (289h49m37.173972537s) [upload=254087]: Got next segment. Inline: false Dec 17 17:18:37.302931863 (289h49m37.174014918s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:18:37.302937634 (289h49m37.174020669s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:18:37.569724064 (289h49m37.440807099s) [seg_pos={0 0}, upload=254087]: Begin upload segment... Dec 17 17:18:37.569770663 (289h49m37.440853708s) [seg_pos={0 0}, upload=254087]: Done begin upload segment. Dec 17 17:18:37.569840656 (289h49m37.440923691s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569858651 (289h49m37.440941696s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569957709 (289h49m37.441040754s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570381687 (289h49m37.441464732s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570721273 (289h49m37.441804319s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571012067 (289h49m37.442095102s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571295546 (289h49m37.442378592s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571934833 (289h49m37.443017878s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572398788 (289h49m37.443481833s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572844126 (289h49m37.443927171s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.573930164 (289h49m37.445013209s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.574335204 (289h49m37.445418249s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575519129 (289h49m37.446602174s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575802970 (289h49m37.446886015s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.578052403 (289h49m37.449135448s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.579787497 (289h49m37.450870542s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:39.276574183 (289h49m39.147657218s) [upload=254087]: Next returned nil segment Dec 17 17:18:39.276593710 (289h49m39.147676765s) [upload=254087]: Waiting for error group managing segments... Dec 17 17:18:39.583499194 (289h49m39.45458224s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.586064850 (289h49m39.457147885s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.627263460 (289h49m39.498346505s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.653542299 (289h49m39.524625355s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.668227403 (289h49m39.539310448s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.776965220 (289h49m39.648048256s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.827312301 (289h49m39.698395336s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.828976920 (289h49m39.700059955s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841154408 (289h49m39.712237453s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841375590 (289h49m39.712458625s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.844226457 (289h49m39.715309492s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.865188934 (289h49m39.736271979s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882207611 (289h49m39.753290656s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882235925 (289h49m39.75331896s) [seg_pos={0 0}, upload=254087]: Segment reached optimal threshold of 13 pieces. Dec 17 17:18:39.882482044 (289h49m39.753565099s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection Dec 17 17:18:39.882491832 (289h49m39.753574877s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882765573 (289h49m39.753848618s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882804698 (289h49m39.753887743s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882826319 (289h49m39.753909485s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882837600 (289h49m39.753920645s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882883669 (289h49m39.753966714s) [seg_pos={0 0}, upload=254087]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:18:39.883006001 (289h49m39.754089056s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:18:39.883050125 (289h49m39.75413317s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:18:40.198144112 (289h49m40.069227157s) [upload=254087]: Done uploading segments... Dec 17 17:18:40.198162958 (289h49m40.069245993s) [upload=254087]: Upload finished. err: <nil> Duration: 18.300954068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................X......................XX..........XX] duration: 30.004043391s trace: https://jaeger.dev.storj.io/trace/379227ab2c9a3c9c FAIL (2025-12-17 17:19:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 17 17:18:37.302512283 (289h49m37.173595329s) [upload=254087]: Starting upload Dec 17 17:18:37.302642772 (289h49m37.173725817s) [upload=254087]: Done starting upload Dec 17 17:18:37.302700041 (289h49m37.173783086s) [upload=254087]: Uploading segments... Dec 17 17:18:37.302889491 (289h49m37.173972537s) [upload=254087]: Got next segment. Inline: false Dec 17 17:18:37.302931863 (289h49m37.174014918s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:18:37.302937634 (289h49m37.174020669s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:18:37.569724064 (289h49m37.440807099s) [seg_pos={0 0}, upload=254087]: Begin upload segment... Dec 17 17:18:37.569770663 (289h49m37.440853708s) [seg_pos={0 0}, upload=254087]: Done begin upload segment. Dec 17 17:18:37.569840656 (289h49m37.440923691s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569858651 (289h49m37.440941696s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.569957709 (289h49m37.441040754s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570381687 (289h49m37.441464732s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.570721273 (289h49m37.441804319s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571012067 (289h49m37.442095102s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571295546 (289h49m37.442378592s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.571934833 (289h49m37.443017878s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572398788 (289h49m37.443481833s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.572844126 (289h49m37.443927171s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.573930164 (289h49m37.445013209s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.574335204 (289h49m37.445418249s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575519129 (289h49m37.446602174s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.575802970 (289h49m37.446886015s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.578052403 (289h49m37.449135448s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:37.579787497 (289h49m37.450870542s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Uploading piece... Dec 17 17:18:39.276574183 (289h49m39.147657218s) [upload=254087]: Next returned nil segment Dec 17 17:18:39.276593710 (289h49m39.147676765s) [upload=254087]: Waiting for error group managing segments... Dec 17 17:18:39.583499194 (289h49m39.45458224s) [noise=true , address=145.79.227.35:4483 , piece_id=SB2VO7VMG6FMHM6KWKSNDPFNVWJYMS67WZDKJYMSW3YPJQSJEEDQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.586064850 (289h49m39.457147885s) [noise=true , address=145.79.227.50:4375 , piece_id=6C26JQA22H3OXYT3OB5RFZAKVLCPC26NENI45ZHKY5DPSXTKNSNA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.627263460 (289h49m39.498346505s) [noise=true , address=145.79.227.28:4726 , piece_id=7XGYTRXFXTDTRE3DHDRE34FRPF3SKTX25G5227S6EMNCU6XCCBEQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.653542299 (289h49m39.524625355s) [noise=true , address=145.79.227.31:4798 , piece_id=PIM4OAUO7Q3O73RD7A2UKNV7LCOV4JIXWP3SKFQJRUOLTRWSZAIQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.668227403 (289h49m39.539310448s) [noise=true , address=145.79.227.29:4943 , piece_id=2E3W2YGS4L4LKQQX5UEEO5HJUZVSDJKATOOK72JADHKI5YKPBWDA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.776965220 (289h49m39.648048256s) [noise=true , address=45.91.116.6:4974 , piece_id=AEJT2GRFMN2ICDETQ2X5SSJJ56264YNUJI2UJFM27N7WDOB7DVKA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.827312301 (289h49m39.698395336s) [noise=true , address=45.91.116.12:4228 , piece_id=GKV7HFIR3PFHVRVLSNL6CDYA4XC5NAVHSO74Z6BV42QFRKP2X2KA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.828976920 (289h49m39.700059955s) [noise=true , address=77.90.31.10:4520 , piece_id=RFACTZQZVBXI6YDQ5VOVWZOEKBOYXHTNAQMTLM4WTMYGO5BM5CFA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841154408 (289h49m39.712237453s) [noise=true , address=45.91.116.10:4082 , piece_id=Q5CV634G27U3YICTLYM5HRKG2PMPXGXKU5P5PNS46E4XXDWFKYPQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.841375590 (289h49m39.712458625s) [noise=true , address=45.91.116.13:4156 , piece_id=Y2ECXHTCEYSPHG6ILUJYFHJOZLE6RSBZKO2EZAZCOPVAMQ4KK4CQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.844226457 (289h49m39.715309492s) [noise=true , address=77.90.31.14:4664 , piece_id=CDIHXZJSTWLOMBN3W347ZLUR5TDUYPHVZD7Z2ONU2G2IYBESZFEA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.865188934 (289h49m39.736271979s) [noise=true , address=45.91.116.8:4046 , piece_id=F7WJI2RXO3IMUYXIMIJYNYPPI7G2PLAVOZSP65WLKEUPL32TCHJA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882207611 (289h49m39.753290656s) [noise=true , address=45.91.116.11:4119 , piece_id=UJG6LGPEZW3RL6PLUCGU4AWUPUC4G4EPO5VHPXKUYCR3P37KOBIA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: <nil> Dec 17 17:18:39.882235925 (289h49m39.75331896s) [seg_pos={0 0}, upload=254087]: Segment reached optimal threshold of 13 pieces. Dec 17 17:18:39.882482044 (289h49m39.753565099s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56256->45.91.116.5:4299: use of closed network connection Dec 17 17:18:39.882491832 (289h49m39.753574877s) [noise=true , address=45.91.116.5:4299 , piece_id=AFTATTURHQ46LXSDIDMEBVLV2EQFZHIBWFAGMODUSF3TF2HWS4AQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882765573 (289h49m39.753848618s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882804698 (289h49m39.753887743s) [noise=true , address=145.79.227.33:4869 , piece_id=QYMI3IB6NCVNZBOLB5RW3VRG7VRTKRNRDJCRR6MO3GARXUS7G4EQ, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882826319 (289h49m39.753909485s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 17 17:18:39.882837600 (289h49m39.753920645s) [noise=true , address=45.91.116.9:4263 , piece_id=PJNGETMQFTBBW2YXR5VAAEMJZGSNMEBQZ3IHNI73SUVM7SCXXUPA, seg_pos={0 0}, upload=254087]: Long tail cancellation due to optimal threshold Dec 17 17:18:39.882883669 (289h49m39.753966714s) [seg_pos={0 0}, upload=254087]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:18:39.883006001 (289h49m39.754089056s) [upload=254087]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:18:39.883050125 (289h49m39.75413317s) [upload=254087]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:18:40.198144112 (289h49m40.069227157s) [upload=254087]: Done uploading segments... Dec 17 17:18:40.198162958 (289h49m40.069245993s) [upload=254087]: Upload finished. err: <nil> Duration: 30.004067969s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 7.247434018s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:19:06 UTC) orbiter warning: context deadline exceeded Duration: 7.247471869s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.XX..XX.....X....X...XX..X..XX...X....................X.....] duration: 3.628766292s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:19:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.630378423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X......................XX..........X] duration: 30.002991313s trace: https://jaeger.dev.storj.io/trace/41aaea217893bfe FAIL (2025-12-17 17:18:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 17 17:17:54.105139009 (289h48m53.976222054s) [upload=254077]: Starting upload Dec 17 17:17:54.105249108 (289h48m53.976332153s) [upload=254077]: Done starting upload Dec 17 17:17:54.105408212 (289h48m53.976491257s) [upload=254077]: Uploading segments... Dec 17 17:17:54.105749080 (289h48m53.976832175s) [upload=254077]: Got next segment. Inline: false Dec 17 17:17:54.105826859 (289h48m53.976909904s) [upload=254077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:17:54.105835265 (289h48m53.97691831s) [upload=254077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:17:54.359975925 (289h48m54.23105897s) [seg_pos={0 0}, upload=254077]: Begin upload segment... Dec 17 17:17:54.360038103 (289h48m54.231121148s) [seg_pos={0 0}, upload=254077]: Done begin upload segment. Dec 17 17:17:54.360090092 (289h48m54.231173127s) [noise=true , address=77.90.31.11:4555 , piece_id=FPNJUMCWBGNRBGDS6NXJJX3BKCORG2QRU4CWV3QHAJPUDJQWBIVA, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.360108868 (289h48m54.231191913s) [noise=true , address=45.91.116.5:4297 , piece_id=BFIM7X4K4NOVDDSS6FKJL4DLN7TF7UOUT27WJUY4UQE5BUNLYVLA, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.360690836 (289h48m54.231773871s) [noise=true , address=45.91.116.10:4082 , piece_id=JXG4CRZ3FRMCE2MN22BZSMPFGSU4O7V3EEVTGKOF7HNE4BRCN3RQ, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.360698120 (289h48m54.231781165s) [noise=true , address=145.79.227.49:4338 , piece_id=WJTWNXLUGAQG4BT5BXJLXRQUZ2SUP4LFXFFD77PIVFZT66LD4SZQ, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.361127888 (289h48m54.232210963s) [noise=true , address=77.90.31.15:4700 , piece_id=DWMUIQNAQVADNWWBFPYCXGUFY6TQIX3JJOGPXKSEVWSQKWG5SS5Q, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.361141945 (289h48m54.232224991s) [noise=true , address=145.79.227.19:4077 , piece_id=D6NXT4XJWXZU2FMAOQUS2M2ROWYWPBX5YMEZ5GGRLUY4KORXCDTQ, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.361678647 (289h48m54.232761682s) [noise=true , address=45.91.116.11:4117 , piece_id=L4KXTWOGGK222KJX6BLMRFFUQ3A5JUX6AQXIJNJOUXVBGNCI4SKQ, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.361871605 (289h48m54.23295464s) [noise=true , address=145.79.227.45:4229 , piece_id=7GNPEYAV2V44ZVFR4QM5GK6X3FJARQ2PTNTG34CBHC3ZYY7T3L6Q, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.362149634 (289h48m54.233232689s) [noise=true , address=45.91.116.9:4261 , piece_id=SJCJOT6KLBQVHRELQ5CKPJFUN26LRSTWSJY7FDNXC4UOPPGZCXEA, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.362590463 (289h48m54.233673508s) [noise=true , address=77.90.31.14:4661 , piece_id=2S2XXNLNYOYB3K3QINSA56VRNXIX2SGQ6VVJEMFEE5YJDSWQUM6Q, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.362778522 (289h48m54.233861567s) [noise=true , address=77.90.31.13:4629 , piece_id=7KL44N55V7VSQ7N5N4V5N74DK2BELH4HPPZVAEELTZLG7XAGBOOA, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.363021625 (289h48m54.23410467s) [noise=true , address=77.90.31.10:4519 , piece_id=YPGERNCWCTQE6P6LAFRKADN3QLSERQ4YNYSDN6DV3WOXTGCDX3VQ, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.363457674 (289h48m54.23454072s) [noise=true , address=145.79.227.34:4906 , piece_id=WSE55QNPFER7YXOL7C4252VSH3S5RX2LOVBHL2YYKT4W6Y5MC4KA, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.363742578 (289h48m54.234825623s) [noise=true , address=45.91.116.12:4227 , piece_id=F2P3MBUSVAZBPGDRF6OOGA5QHAYIFKI2QT6PM4QTUNEBE4BGNYKQ, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.364093686 (289h48m54.235176721s) [noise=true , address=45.91.116.6:4974 , piece_id=6WE3RROW636QHZX7VGSH6GDQAADES2FENIRA6HRDW6ADXB7KGJLQ, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:54.364415940 (289h48m54.235499005s) [noise=true , address=145.79.227.52:4446 , piece_id=RS6GDW6KDJLYBZJX7BWRMFSFQN5YHS6R5TYZZ4MF4RNBWLXOCE3A, seg_pos={0 0}, upload=254077]: Uploading piece... Dec 17 17:17:56.026887032 (289h48m55.897970067s) [upload=254077]: Next returned nil segment Dec 17 17:17:56.026904094 (289h48m55.897987139s) [upload=254077]: Waiting for error group managing segments... Dec 17 17:17:56.345599048 (289h48m56.216682093s) [noise=true , address=145.79.227.34:4906 , piece_id=WSE55QNPFER7YXOL7C4252VSH3S5RX2LOVBHL2YYKT4W6Y5MC4KA, seg_pos={0 0}, upload=254077]: Done uploading piece. err: <nil> Dec 17 17:17:56.362542061 (289h48m56.233625096s) [noise=true , address=145.79.227.45:4229 , piece_id=7GNPEYAV2V44ZVFR4QM5GK6X3FJARQ2PTNTG34CBHC3ZYY7T3L6Q, seg_pos={0 0}, upload=254077]: Done uploading piece. err: <nil> Dec 17 17:17:56.367467500 (289h48m56.238550545s) [noise=true , address=145.79.227.49:4338 , piece_id=WJTWNXLUGAQG4BT5BXJLXRQUZ2SUP4LFXFFD77PIVFZT66LD4SZQ, seg_pos={0 0}, upload=254077]: Done uploading piece. err: <nil> Dec 17 17:17:56.374804522 (289h48m56.245887567s) [noise=true , address=145.79.227.19:4077 , piece_id=D6NXT4XJWXZU2FMAOQUS2M2ROWYWPBX5YMEZ5GGRLUY4KORXCDTQ, seg_pos={0 0}, upload=254077]: Done uploading piece. err: <nil> Dec 17 17:17:56.375027295 (289h48m56.24611034s) [noise=true , address=145.79.227.52:4446 , piece_id=RS6GDW6KDJLYBZJX7BWRMFSFQN5YHS6R5TYZZ4MF4RNBWLXOCE3A, seg_pos={0 0}, upload=254077]: Done uploading piece. err: <nil> Dec 17 17:17:56.516649240 (289h48m56.387732275s) [noise=true , address=77.90.31.11:4555 , piece_id=FPNJUMCWBGNRBGDS6NXJJX3BKCORG2QRU4CWV3QHAJPUDJQWBIVA, seg_pos={0 0}, upload=254077]: Done uploading piece. err: <nil> Dec 17 17:17:56.543700761 (289h48m56.414783806s) [noise=true , address=77.90.31.10:4519 , piece_id=YPGERNCWCTQE6P6LAFRKADN3QLSERQ4YNYSDN6DV3WOXTGCDX3VQ, seg_pos={0 0}, upload=254077]: Done uploading piece. err: <nil> Dec 17 17:17:56.590955092 (289h48m56.462038137s) [noise=true , address=77.90.31.14:4661 , piece_id=2S2XXNLNYOYB3K3QINSA56VRNXIX2SGQ6VVJEMFEE5YJDSWQUM6Q, seg_pos={0 0}, upload=254077]: Done uploading piece. err: <nil> Dec 17 17:17:56.609138055 (289h48m56.4802211s) [noise=true , address=45.91.116.5:4297 , piece_id=BFIM7X4K4NOVDDSS6FKJL4DLN7TF7UOUT27WJUY4UQE5BUNLYVLA, seg_pos={0 0}, upload=254077]: Done uploading piece. err: <nil> Dec 17 17:17:56.616325542 (289h48m56.487408587s) [noise=true , address=45.91.116.10:4082 , piece_id=JXG4CRZ3FRMCE2MN22BZSMPFGSU4O7V3EEVTGKOF7HNE4BRCN3RQ, seg_pos={0 0}, upload=254077]: Done uploading piece. err: <nil> Dec 17 17:17:56.619814045 (289h48m56.49089708s) [noise=true , address=45.91.116.12:4227 , piece_id=F2P3MBUSVAZBPGDRF6OOGA5QHAYIFKI2QT6PM4QTUNEBE4BGNYKQ, seg_pos={0 0}, upload=254077]: Done uploading piece. err: <nil> Dec 17 17:17:56.658094057 (289h48m56.529177102s) [noise=true , address=45.91.116.6:4974 , piece_id=6WE3RROW636QHZX7VGSH6GDQAADES2FENIRA6HRDW6ADXB7KGJLQ, seg_pos={0 0}, upload=254077]: Done uploading piece. err: <nil> Dec 17 17:17:56.659607229 (289h48m56.530690274s) [noise=true , address=77.90.31.13:4629 , piece_id=7KL44N55V7VSQ7N5N4V5N74DK2BELH4HPPZVAEELTZLG7XAGBOOA, seg_pos={0 0}, upload=254077]: Done uploading piece. err: <nil> Dec 17 17:17:56.659625753 (289h48m56.530708798s) [seg_pos={0 0}, upload=254077]: Segment reached optimal threshold of 13 pieces. Dec 17 17:17:56.659844210 (289h48m56.530927245s) [noise=true , address=77.90.31.15:4700 , piece_id=DWMUIQNAQVADNWWBFPYCXGUFY6TQIX3JJOGPXKSEVWSQKWG5SS5Q, seg_pos={0 0}, upload=254077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 17 17:17:56.659853998 (289h48m56.530937043s) [noise=true , address=77.90.31.15:4700 , piece_id=DWMUIQNAQVADNWWBFPYCXGUFY6TQIX3JJOGPXKSEVWSQKWG5SS5Q, seg_pos={0 0}, upload=254077]: Long tail cancellation due to optimal threshold Dec 17 17:17:56.660164189 (289h48m56.531247244s) [noise=true , address=45.91.116.11:4117 , piece_id=L4KXTWOGGK222KJX6BLMRFFUQ3A5JUX6AQXIJNJOUXVBGNCI4SKQ, seg_pos={0 0}, upload=254077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 17 17:17:56.660175110 (289h48m56.531258145s) [noise=true , address=45.91.116.11:4117 , piece_id=L4KXTWOGGK222KJX6BLMRFFUQ3A5JUX6AQXIJNJOUXVBGNCI4SKQ, seg_pos={0 0}, upload=254077]: Long tail cancellation due to optimal threshold Dec 17 17:17:56.660315337 (289h48m56.531398382s) [noise=true , address=45.91.116.9:4261 , piece_id=SJCJOT6KLBQVHRELQ5CKPJFUN26LRSTWSJY7FDNXC4UOPPGZCXEA, seg_pos={0 0}, upload=254077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 17 17:17:56.660323211 (289h48m56.531406256s) [noise=true , address=45.91.116.9:4261 , piece_id=SJCJOT6KLBQVHRELQ5CKPJFUN26LRSTWSJY7FDNXC4UOPPGZCXEA, seg_pos={0 0}, upload=254077]: Long tail cancellation due to optimal threshold Dec 17 17:17:56.660351185 (289h48m56.53143422s) [seg_pos={0 0}, upload=254077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:17:56.660436658 (289h48m56.531519703s) [upload=254077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:17:56.660443411 (289h48m56.531526456s) [upload=254077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:17:56.956764481 (289h48m56.827847536s) [upload=254077]: Done uploading segments... Dec 17 17:17:56.956789308 (289h48m56.827872354s) [upload=254077]: Upload finished. err: <nil> Duration: 30.00301633s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 7.071568314s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:18:08 UTC) orbiter warning: context deadline exceeded Duration: 7.071601336s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X.XX..XX.....X....X...XX..X..XX...X....................X....] duration: 3.597128839s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:17:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.597155249s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XX.XX..XX.....X....X...XX..X..XX...X....................X...] duration: 5.001103478s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:17:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001143914s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 14.31179753s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:17:01 UTC) orbiter warning: context deadline exceeded Duration: 14.311904584s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X......................XX..........] duration: 5.934297333s trace: https://jaeger.dev.storj.io/trace/435a94c4a7186daa FAIL (2025-12-17 17:16:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ) context deadline exceeded ======LOGS====== Dec 17 17:08:05.711119168 (289h39m5.582202214s) [upload=253937]: Starting upload Dec 17 17:08:05.711211945 (289h39m5.58229498s) [upload=253937]: Done starting upload Dec 17 17:08:05.711475638 (289h39m5.582558683s) [upload=253937]: Uploading segments... Dec 17 17:08:05.711534189 (289h39m5.582617244s) [upload=253937]: Got next segment. Inline: false Dec 17 17:08:05.711575557 (289h39m5.582658602s) [upload=253937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:08:05.711581730 (289h39m5.582664775s) [upload=253937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:08:05.961618917 (289h39m5.832701952s) [seg_pos={0 0}, upload=253937]: Begin upload segment... Dec 17 17:08:05.961679182 (289h39m5.832762227s) [seg_pos={0 0}, upload=253937]: Done begin upload segment. Dec 17 17:08:05.961761188 (289h39m5.832844243s) [noise=true , address=145.79.227.43:4158 , piece_id=TURS2YMHYW23WZGMIYZ5YFVYFQYBKJKDZTRPMVR7U4APIAFZNAEA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.961839697 (289h39m5.832922742s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.962527207 (289h39m5.833610252s) [noise=true , address=45.91.116.14:4190 , piece_id=AWTFBZTNXUM4N2UO7OFMENKYF7YMFTEIQWIIB2DZKBFSG5KTFBMQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.962964238 (289h39m5.834047283s) [noise=true , address=145.79.227.13:4803 , piece_id=FUWPMH3PGYS64TDV2YZLARRYO2Q3MZTGJKCPCXLXGAOLZRFWNYTA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.963481283 (289h39m5.834564338s) [noise=true , address=45.91.116.12:4226 , piece_id=DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.963977248 (289h39m5.835060293s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.964461880 (289h39m5.835544925s) [noise=true , address=145.79.227.34:4907 , piece_id=D3NCY5V32N7AP23QEZYIU5GOKWCX5A4DMYMRYWTMSX3OJ6JETJNA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.964959188 (289h39m5.836042233s) [noise=true , address=145.79.227.47:4266 , piece_id=BDPKINHG3BEUUYUMWK323J3HQOPOZBZNMACVMWDZ6URRGEG5ZV7A, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.965479038 (289h39m5.836562083s) [noise=true , address=145.79.227.36:4517 , piece_id=7PKDM7LC3PDZ33DZY5AM3TQOI25WVGUFNAWL6TSOXQ5Y52VSTBRA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.966053180 (289h39m5.837136225s) [noise=true , address=145.79.227.48:4303 , piece_id=EH6WZNR7PU5B3DIVMN5LRHDIJXEU5P7SN4XHVER6JK74LYW5PM7Q, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.966599952 (289h39m5.837682997s) [noise=true , address=145.79.227.46:4601 , piece_id=OQF64QD2T3F7Q2ZHYOF2PBY7T3SXBJE76Q4XVZFNPEDGMXPIVJ6Q, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.967138547 (289h39m5.838221582s) [noise=true , address=145.79.227.11:4730 , piece_id=ZVC3HDEOMANUTXQ4ORLFH3SDNJ3HRLVRVH6GHEKDAFYOZQK2OOYA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.967402069 (289h39m5.838485114s) [noise=true , address=145.79.227.18:4006 , piece_id=ZX5SWPZWZSPEGEZB75LJGS6OQAEZQQPOUTAYG6HTADXXJBT75ZXQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.968131428 (289h39m5.839214473s) [noise=true , address=77.90.31.21:4877 , piece_id=WTCYZ7PKCT27JZDXSLXQ7SPAPO5OBBOECVKRMULKB7WWJT7GXENA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.968831491 (289h39m5.839914536s) [noise=true , address=45.91.116.11:4117 , piece_id=BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.969248144 (289h39m5.840331189s) [noise=true , address=145.79.227.10:4402 , piece_id=S4K3TT37OZSDS4KDOC4YAALVXUTY2IHUKXQGS5R3OLSD6624S3MQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:07.095691332 (289h39m6.966774377s) [noise=true , address=77.90.31.21:4877 , piece_id=WTCYZ7PKCT27JZDXSLXQ7SPAPO5OBBOECVKRMULKB7WWJT7GXENA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 17:08:07.677933289 (289h39m7.549016334s) [upload=253937]: Next returned nil segment Dec 17 17:08:07.677950371 (289h39m7.549033406s) [upload=253937]: Waiting for error group managing segments... Dec 17 17:08:07.958796824 (289h39m7.829879869s) [noise=true , address=145.79.227.36:4517 , piece_id=7PKDM7LC3PDZ33DZY5AM3TQOI25WVGUFNAWL6TSOXQ5Y52VSTBRA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.967087221 (289h39m7.838170266s) [noise=true , address=145.79.227.10:4402 , piece_id=S4K3TT37OZSDS4KDOC4YAALVXUTY2IHUKXQGS5R3OLSD6624S3MQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.968141288 (289h39m7.839224333s) [noise=true , address=145.79.227.47:4266 , piece_id=BDPKINHG3BEUUYUMWK323J3HQOPOZBZNMACVMWDZ6URRGEG5ZV7A, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.974117077 (289h39m7.845200113s) [noise=true , address=145.79.227.46:4601 , piece_id=OQF64QD2T3F7Q2ZHYOF2PBY7T3SXBJE76Q4XVZFNPEDGMXPIVJ6Q, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.974188333 (289h39m7.845271378s) [noise=true , address=145.79.227.18:4006 , piece_id=ZX5SWPZWZSPEGEZB75LJGS6OQAEZQQPOUTAYG6HTADXXJBT75ZXQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.977968792 (289h39m7.849051837s) [noise=true , address=145.79.227.34:4907 , piece_id=D3NCY5V32N7AP23QEZYIU5GOKWCX5A4DMYMRYWTMSX3OJ6JETJNA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.980381085 (289h39m7.85146413s) [noise=true , address=145.79.227.13:4803 , piece_id=FUWPMH3PGYS64TDV2YZLARRYO2Q3MZTGJKCPCXLXGAOLZRFWNYTA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.026636303 (289h39m7.897719348s) [noise=true , address=145.79.227.43:4158 , piece_id=TURS2YMHYW23WZGMIYZ5YFVYFQYBKJKDZTRPMVR7U4APIAFZNAEA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.027057596 (289h39m7.898140641s) [noise=true , address=145.79.227.11:4730 , piece_id=ZVC3HDEOMANUTXQ4ORLFH3SDNJ3HRLVRVH6GHEKDAFYOZQK2OOYA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.046147687 (289h39m7.917230732s) [noise=true , address=145.79.227.48:4303 , piece_id=EH6WZNR7PU5B3DIVMN5LRHDIJXEU5P7SN4XHVER6JK74LYW5PM7Q, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.264192205 (289h39m8.13527525s) [noise=true , address=45.91.116.11:4117 , piece_id=BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.514532037 (289h39m8.385615083s) [noise=true , address=45.91.116.12:4226 , piece_id=DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.581684037 (289h39m8.452767082s) [noise=true , address=45.91.116.14:4190 , piece_id=AWTFBZTNXUM4N2UO7OFMENKYF7YMFTEIQWIIB2DZKBFSG5KTFBMQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.581708444 (289h39m8.452791489s) [seg_pos={0 0}, upload=253937]: Segment reached optimal threshold of 13 pieces. Dec 17 17:08:08.581998486 (289h39m8.453081521s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 17 17:08:08.582014847 (289h39m8.453097882s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Long tail cancellation due to optimal threshold Dec 17 17:08:08.582124086 (289h39m8.453207141s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 17 17:08:08.582133874 (289h39m8.453216909s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Long tail cancellation due to optimal threshold Dec 17 17:08:08.582169963 (289h39m8.453253008s) [seg_pos={0 0}, upload=253937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:08:08.582283388 (289h39m8.453366444s) [upload=253937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:08:08.582292756 (289h39m8.453375801s) [upload=253937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:08:08.873888509 (289h39m8.744971554s) [upload=253937]: Done uploading segments... Dec 17 17:08:08.873913225 (289h39m8.74499628s) [upload=253937]: Upload finished. err: <nil> Duration: 5.934323513s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [XXX.XX..XX.....X....X...XX..X..XX...X....................X..] duration: 3.54587847s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:16:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.545908417s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.448635003s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:16:08 UTC) orbiter warning: context deadline exceeded Duration: 6.448665521s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X......................XX.........] duration: 6.701887775s trace: https://jaeger.dev.storj.io/trace/435a94c4a7186daa FAIL (2025-12-17 17:15:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ) context deadline exceeded ======LOGS====== Dec 17 17:08:05.711119168 (289h39m5.582202214s) [upload=253937]: Starting upload Dec 17 17:08:05.711211945 (289h39m5.58229498s) [upload=253937]: Done starting upload Dec 17 17:08:05.711475638 (289h39m5.582558683s) [upload=253937]: Uploading segments... Dec 17 17:08:05.711534189 (289h39m5.582617244s) [upload=253937]: Got next segment. Inline: false Dec 17 17:08:05.711575557 (289h39m5.582658602s) [upload=253937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:08:05.711581730 (289h39m5.582664775s) [upload=253937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:08:05.961618917 (289h39m5.832701952s) [seg_pos={0 0}, upload=253937]: Begin upload segment... Dec 17 17:08:05.961679182 (289h39m5.832762227s) [seg_pos={0 0}, upload=253937]: Done begin upload segment. Dec 17 17:08:05.961761188 (289h39m5.832844243s) [noise=true , address=145.79.227.43:4158 , piece_id=TURS2YMHYW23WZGMIYZ5YFVYFQYBKJKDZTRPMVR7U4APIAFZNAEA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.961839697 (289h39m5.832922742s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.962527207 (289h39m5.833610252s) [noise=true , address=45.91.116.14:4190 , piece_id=AWTFBZTNXUM4N2UO7OFMENKYF7YMFTEIQWIIB2DZKBFSG5KTFBMQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.962964238 (289h39m5.834047283s) [noise=true , address=145.79.227.13:4803 , piece_id=FUWPMH3PGYS64TDV2YZLARRYO2Q3MZTGJKCPCXLXGAOLZRFWNYTA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.963481283 (289h39m5.834564338s) [noise=true , address=45.91.116.12:4226 , piece_id=DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.963977248 (289h39m5.835060293s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.964461880 (289h39m5.835544925s) [noise=true , address=145.79.227.34:4907 , piece_id=D3NCY5V32N7AP23QEZYIU5GOKWCX5A4DMYMRYWTMSX3OJ6JETJNA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.964959188 (289h39m5.836042233s) [noise=true , address=145.79.227.47:4266 , piece_id=BDPKINHG3BEUUYUMWK323J3HQOPOZBZNMACVMWDZ6URRGEG5ZV7A, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.965479038 (289h39m5.836562083s) [noise=true , address=145.79.227.36:4517 , piece_id=7PKDM7LC3PDZ33DZY5AM3TQOI25WVGUFNAWL6TSOXQ5Y52VSTBRA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.966053180 (289h39m5.837136225s) [noise=true , address=145.79.227.48:4303 , piece_id=EH6WZNR7PU5B3DIVMN5LRHDIJXEU5P7SN4XHVER6JK74LYW5PM7Q, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.966599952 (289h39m5.837682997s) [noise=true , address=145.79.227.46:4601 , piece_id=OQF64QD2T3F7Q2ZHYOF2PBY7T3SXBJE76Q4XVZFNPEDGMXPIVJ6Q, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.967138547 (289h39m5.838221582s) [noise=true , address=145.79.227.11:4730 , piece_id=ZVC3HDEOMANUTXQ4ORLFH3SDNJ3HRLVRVH6GHEKDAFYOZQK2OOYA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.967402069 (289h39m5.838485114s) [noise=true , address=145.79.227.18:4006 , piece_id=ZX5SWPZWZSPEGEZB75LJGS6OQAEZQQPOUTAYG6HTADXXJBT75ZXQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.968131428 (289h39m5.839214473s) [noise=true , address=77.90.31.21:4877 , piece_id=WTCYZ7PKCT27JZDXSLXQ7SPAPO5OBBOECVKRMULKB7WWJT7GXENA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.968831491 (289h39m5.839914536s) [noise=true , address=45.91.116.11:4117 , piece_id=BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.969248144 (289h39m5.840331189s) [noise=true , address=145.79.227.10:4402 , piece_id=S4K3TT37OZSDS4KDOC4YAALVXUTY2IHUKXQGS5R3OLSD6624S3MQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:07.095691332 (289h39m6.966774377s) [noise=true , address=77.90.31.21:4877 , piece_id=WTCYZ7PKCT27JZDXSLXQ7SPAPO5OBBOECVKRMULKB7WWJT7GXENA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 17:08:07.677933289 (289h39m7.549016334s) [upload=253937]: Next returned nil segment Dec 17 17:08:07.677950371 (289h39m7.549033406s) [upload=253937]: Waiting for error group managing segments... Dec 17 17:08:07.958796824 (289h39m7.829879869s) [noise=true , address=145.79.227.36:4517 , piece_id=7PKDM7LC3PDZ33DZY5AM3TQOI25WVGUFNAWL6TSOXQ5Y52VSTBRA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.967087221 (289h39m7.838170266s) [noise=true , address=145.79.227.10:4402 , piece_id=S4K3TT37OZSDS4KDOC4YAALVXUTY2IHUKXQGS5R3OLSD6624S3MQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.968141288 (289h39m7.839224333s) [noise=true , address=145.79.227.47:4266 , piece_id=BDPKINHG3BEUUYUMWK323J3HQOPOZBZNMACVMWDZ6URRGEG5ZV7A, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.974117077 (289h39m7.845200113s) [noise=true , address=145.79.227.46:4601 , piece_id=OQF64QD2T3F7Q2ZHYOF2PBY7T3SXBJE76Q4XVZFNPEDGMXPIVJ6Q, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.974188333 (289h39m7.845271378s) [noise=true , address=145.79.227.18:4006 , piece_id=ZX5SWPZWZSPEGEZB75LJGS6OQAEZQQPOUTAYG6HTADXXJBT75ZXQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.977968792 (289h39m7.849051837s) [noise=true , address=145.79.227.34:4907 , piece_id=D3NCY5V32N7AP23QEZYIU5GOKWCX5A4DMYMRYWTMSX3OJ6JETJNA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.980381085 (289h39m7.85146413s) [noise=true , address=145.79.227.13:4803 , piece_id=FUWPMH3PGYS64TDV2YZLARRYO2Q3MZTGJKCPCXLXGAOLZRFWNYTA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.026636303 (289h39m7.897719348s) [noise=true , address=145.79.227.43:4158 , piece_id=TURS2YMHYW23WZGMIYZ5YFVYFQYBKJKDZTRPMVR7U4APIAFZNAEA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.027057596 (289h39m7.898140641s) [noise=true , address=145.79.227.11:4730 , piece_id=ZVC3HDEOMANUTXQ4ORLFH3SDNJ3HRLVRVH6GHEKDAFYOZQK2OOYA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.046147687 (289h39m7.917230732s) [noise=true , address=145.79.227.48:4303 , piece_id=EH6WZNR7PU5B3DIVMN5LRHDIJXEU5P7SN4XHVER6JK74LYW5PM7Q, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.264192205 (289h39m8.13527525s) [noise=true , address=45.91.116.11:4117 , piece_id=BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.514532037 (289h39m8.385615083s) [noise=true , address=45.91.116.12:4226 , piece_id=DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.581684037 (289h39m8.452767082s) [noise=true , address=45.91.116.14:4190 , piece_id=AWTFBZTNXUM4N2UO7OFMENKYF7YMFTEIQWIIB2DZKBFSG5KTFBMQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.581708444 (289h39m8.452791489s) [seg_pos={0 0}, upload=253937]: Segment reached optimal threshold of 13 pieces. Dec 17 17:08:08.581998486 (289h39m8.453081521s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 17 17:08:08.582014847 (289h39m8.453097882s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Long tail cancellation due to optimal threshold Dec 17 17:08:08.582124086 (289h39m8.453207141s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 17 17:08:08.582133874 (289h39m8.453216909s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Long tail cancellation due to optimal threshold Dec 17 17:08:08.582169963 (289h39m8.453253008s) [seg_pos={0 0}, upload=253937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:08:08.582283388 (289h39m8.453366444s) [upload=253937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:08:08.582292756 (289h39m8.453375801s) [upload=253937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:08:08.873888509 (289h39m8.744971554s) [upload=253937]: Done uploading segments... Dec 17 17:08:08.873913225 (289h39m8.74499628s) [upload=253937]: Upload finished. err: <nil> Duration: 6.701903976s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [XXXX.XX..XX.....X....X...XX..X..XX...X....................X.] duration: 3.536770146s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:15:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.536802316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X......................XX........] duration: 29.51604443s trace: https://jaeger.dev.storj.io/trace/435a94c4a7186daa FAIL (2025-12-17 17:15:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ) context deadline exceeded ======LOGS====== Dec 17 17:08:05.711119168 (289h39m5.582202214s) [upload=253937]: Starting upload Dec 17 17:08:05.711211945 (289h39m5.58229498s) [upload=253937]: Done starting upload Dec 17 17:08:05.711475638 (289h39m5.582558683s) [upload=253937]: Uploading segments... Dec 17 17:08:05.711534189 (289h39m5.582617244s) [upload=253937]: Got next segment. Inline: false Dec 17 17:08:05.711575557 (289h39m5.582658602s) [upload=253937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:08:05.711581730 (289h39m5.582664775s) [upload=253937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:08:05.961618917 (289h39m5.832701952s) [seg_pos={0 0}, upload=253937]: Begin upload segment... Dec 17 17:08:05.961679182 (289h39m5.832762227s) [seg_pos={0 0}, upload=253937]: Done begin upload segment. Dec 17 17:08:05.961761188 (289h39m5.832844243s) [noise=true , address=145.79.227.43:4158 , piece_id=TURS2YMHYW23WZGMIYZ5YFVYFQYBKJKDZTRPMVR7U4APIAFZNAEA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.961839697 (289h39m5.832922742s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.962527207 (289h39m5.833610252s) [noise=true , address=45.91.116.14:4190 , piece_id=AWTFBZTNXUM4N2UO7OFMENKYF7YMFTEIQWIIB2DZKBFSG5KTFBMQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.962964238 (289h39m5.834047283s) [noise=true , address=145.79.227.13:4803 , piece_id=FUWPMH3PGYS64TDV2YZLARRYO2Q3MZTGJKCPCXLXGAOLZRFWNYTA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.963481283 (289h39m5.834564338s) [noise=true , address=45.91.116.12:4226 , piece_id=DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.963977248 (289h39m5.835060293s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.964461880 (289h39m5.835544925s) [noise=true , address=145.79.227.34:4907 , piece_id=D3NCY5V32N7AP23QEZYIU5GOKWCX5A4DMYMRYWTMSX3OJ6JETJNA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.964959188 (289h39m5.836042233s) [noise=true , address=145.79.227.47:4266 , piece_id=BDPKINHG3BEUUYUMWK323J3HQOPOZBZNMACVMWDZ6URRGEG5ZV7A, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.965479038 (289h39m5.836562083s) [noise=true , address=145.79.227.36:4517 , piece_id=7PKDM7LC3PDZ33DZY5AM3TQOI25WVGUFNAWL6TSOXQ5Y52VSTBRA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.966053180 (289h39m5.837136225s) [noise=true , address=145.79.227.48:4303 , piece_id=EH6WZNR7PU5B3DIVMN5LRHDIJXEU5P7SN4XHVER6JK74LYW5PM7Q, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.966599952 (289h39m5.837682997s) [noise=true , address=145.79.227.46:4601 , piece_id=OQF64QD2T3F7Q2ZHYOF2PBY7T3SXBJE76Q4XVZFNPEDGMXPIVJ6Q, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.967138547 (289h39m5.838221582s) [noise=true , address=145.79.227.11:4730 , piece_id=ZVC3HDEOMANUTXQ4ORLFH3SDNJ3HRLVRVH6GHEKDAFYOZQK2OOYA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.967402069 (289h39m5.838485114s) [noise=true , address=145.79.227.18:4006 , piece_id=ZX5SWPZWZSPEGEZB75LJGS6OQAEZQQPOUTAYG6HTADXXJBT75ZXQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.968131428 (289h39m5.839214473s) [noise=true , address=77.90.31.21:4877 , piece_id=WTCYZ7PKCT27JZDXSLXQ7SPAPO5OBBOECVKRMULKB7WWJT7GXENA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.968831491 (289h39m5.839914536s) [noise=true , address=45.91.116.11:4117 , piece_id=BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.969248144 (289h39m5.840331189s) [noise=true , address=145.79.227.10:4402 , piece_id=S4K3TT37OZSDS4KDOC4YAALVXUTY2IHUKXQGS5R3OLSD6624S3MQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:07.095691332 (289h39m6.966774377s) [noise=true , address=77.90.31.21:4877 , piece_id=WTCYZ7PKCT27JZDXSLXQ7SPAPO5OBBOECVKRMULKB7WWJT7GXENA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 17:08:07.677933289 (289h39m7.549016334s) [upload=253937]: Next returned nil segment Dec 17 17:08:07.677950371 (289h39m7.549033406s) [upload=253937]: Waiting for error group managing segments... Dec 17 17:08:07.958796824 (289h39m7.829879869s) [noise=true , address=145.79.227.36:4517 , piece_id=7PKDM7LC3PDZ33DZY5AM3TQOI25WVGUFNAWL6TSOXQ5Y52VSTBRA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.967087221 (289h39m7.838170266s) [noise=true , address=145.79.227.10:4402 , piece_id=S4K3TT37OZSDS4KDOC4YAALVXUTY2IHUKXQGS5R3OLSD6624S3MQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.968141288 (289h39m7.839224333s) [noise=true , address=145.79.227.47:4266 , piece_id=BDPKINHG3BEUUYUMWK323J3HQOPOZBZNMACVMWDZ6URRGEG5ZV7A, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.974117077 (289h39m7.845200113s) [noise=true , address=145.79.227.46:4601 , piece_id=OQF64QD2T3F7Q2ZHYOF2PBY7T3SXBJE76Q4XVZFNPEDGMXPIVJ6Q, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.974188333 (289h39m7.845271378s) [noise=true , address=145.79.227.18:4006 , piece_id=ZX5SWPZWZSPEGEZB75LJGS6OQAEZQQPOUTAYG6HTADXXJBT75ZXQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.977968792 (289h39m7.849051837s) [noise=true , address=145.79.227.34:4907 , piece_id=D3NCY5V32N7AP23QEZYIU5GOKWCX5A4DMYMRYWTMSX3OJ6JETJNA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.980381085 (289h39m7.85146413s) [noise=true , address=145.79.227.13:4803 , piece_id=FUWPMH3PGYS64TDV2YZLARRYO2Q3MZTGJKCPCXLXGAOLZRFWNYTA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.026636303 (289h39m7.897719348s) [noise=true , address=145.79.227.43:4158 , piece_id=TURS2YMHYW23WZGMIYZ5YFVYFQYBKJKDZTRPMVR7U4APIAFZNAEA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.027057596 (289h39m7.898140641s) [noise=true , address=145.79.227.11:4730 , piece_id=ZVC3HDEOMANUTXQ4ORLFH3SDNJ3HRLVRVH6GHEKDAFYOZQK2OOYA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.046147687 (289h39m7.917230732s) [noise=true , address=145.79.227.48:4303 , piece_id=EH6WZNR7PU5B3DIVMN5LRHDIJXEU5P7SN4XHVER6JK74LYW5PM7Q, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.264192205 (289h39m8.13527525s) [noise=true , address=45.91.116.11:4117 , piece_id=BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.514532037 (289h39m8.385615083s) [noise=true , address=45.91.116.12:4226 , piece_id=DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.581684037 (289h39m8.452767082s) [noise=true , address=45.91.116.14:4190 , piece_id=AWTFBZTNXUM4N2UO7OFMENKYF7YMFTEIQWIIB2DZKBFSG5KTFBMQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.581708444 (289h39m8.452791489s) [seg_pos={0 0}, upload=253937]: Segment reached optimal threshold of 13 pieces. Dec 17 17:08:08.581998486 (289h39m8.453081521s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 17 17:08:08.582014847 (289h39m8.453097882s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Long tail cancellation due to optimal threshold Dec 17 17:08:08.582124086 (289h39m8.453207141s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 17 17:08:08.582133874 (289h39m8.453216909s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Long tail cancellation due to optimal threshold Dec 17 17:08:08.582169963 (289h39m8.453253008s) [seg_pos={0 0}, upload=253937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:08:08.582283388 (289h39m8.453366444s) [upload=253937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:08:08.582292756 (289h39m8.453375801s) [upload=253937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:08:08.873888509 (289h39m8.744971554s) [upload=253937]: Done uploading segments... Dec 17 17:08:08.873913225 (289h39m8.74499628s) [upload=253937]: Upload finished. err: <nil> Duration: 29.516074858s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 10.479368526s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:15:05 UTC) orbiter warning: context deadline exceeded Duration: 10.479399655s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [XXXXX.XX..XX.....X....X...XX..X..XX...X....................X] duration: 5.001693621s trace: https://jaeger.dev.storj.io/trace/3b543b50c485005c FAIL (2025-12-17 17:14:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00176605s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [XXXXXX.XX..XX.....X....X...XX..X..XX...X....................] duration: 3.465037301s trace: https://jaeger.dev.storj.io/trace/79887e0b0ab4d774 FAIL (2025-12-17 17:14:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46506244s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 10.601655664s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:14:03 UTC) orbiter warning: context deadline exceeded Duration: 10.601674849s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X......................XX.......] duration: 5.752546684s trace: https://jaeger.dev.storj.io/trace/435a94c4a7186daa FAIL (2025-12-17 17:13:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ) context deadline exceeded ======LOGS====== Dec 17 17:08:05.711119168 (289h39m5.582202214s) [upload=253937]: Starting upload Dec 17 17:08:05.711211945 (289h39m5.58229498s) [upload=253937]: Done starting upload Dec 17 17:08:05.711475638 (289h39m5.582558683s) [upload=253937]: Uploading segments... Dec 17 17:08:05.711534189 (289h39m5.582617244s) [upload=253937]: Got next segment. Inline: false Dec 17 17:08:05.711575557 (289h39m5.582658602s) [upload=253937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:08:05.711581730 (289h39m5.582664775s) [upload=253937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:08:05.961618917 (289h39m5.832701952s) [seg_pos={0 0}, upload=253937]: Begin upload segment... Dec 17 17:08:05.961679182 (289h39m5.832762227s) [seg_pos={0 0}, upload=253937]: Done begin upload segment. Dec 17 17:08:05.961761188 (289h39m5.832844243s) [noise=true , address=145.79.227.43:4158 , piece_id=TURS2YMHYW23WZGMIYZ5YFVYFQYBKJKDZTRPMVR7U4APIAFZNAEA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.961839697 (289h39m5.832922742s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.962527207 (289h39m5.833610252s) [noise=true , address=45.91.116.14:4190 , piece_id=AWTFBZTNXUM4N2UO7OFMENKYF7YMFTEIQWIIB2DZKBFSG5KTFBMQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.962964238 (289h39m5.834047283s) [noise=true , address=145.79.227.13:4803 , piece_id=FUWPMH3PGYS64TDV2YZLARRYO2Q3MZTGJKCPCXLXGAOLZRFWNYTA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.963481283 (289h39m5.834564338s) [noise=true , address=45.91.116.12:4226 , piece_id=DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.963977248 (289h39m5.835060293s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.964461880 (289h39m5.835544925s) [noise=true , address=145.79.227.34:4907 , piece_id=D3NCY5V32N7AP23QEZYIU5GOKWCX5A4DMYMRYWTMSX3OJ6JETJNA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.964959188 (289h39m5.836042233s) [noise=true , address=145.79.227.47:4266 , piece_id=BDPKINHG3BEUUYUMWK323J3HQOPOZBZNMACVMWDZ6URRGEG5ZV7A, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.965479038 (289h39m5.836562083s) [noise=true , address=145.79.227.36:4517 , piece_id=7PKDM7LC3PDZ33DZY5AM3TQOI25WVGUFNAWL6TSOXQ5Y52VSTBRA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.966053180 (289h39m5.837136225s) [noise=true , address=145.79.227.48:4303 , piece_id=EH6WZNR7PU5B3DIVMN5LRHDIJXEU5P7SN4XHVER6JK74LYW5PM7Q, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.966599952 (289h39m5.837682997s) [noise=true , address=145.79.227.46:4601 , piece_id=OQF64QD2T3F7Q2ZHYOF2PBY7T3SXBJE76Q4XVZFNPEDGMXPIVJ6Q, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.967138547 (289h39m5.838221582s) [noise=true , address=145.79.227.11:4730 , piece_id=ZVC3HDEOMANUTXQ4ORLFH3SDNJ3HRLVRVH6GHEKDAFYOZQK2OOYA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.967402069 (289h39m5.838485114s) [noise=true , address=145.79.227.18:4006 , piece_id=ZX5SWPZWZSPEGEZB75LJGS6OQAEZQQPOUTAYG6HTADXXJBT75ZXQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.968131428 (289h39m5.839214473s) [noise=true , address=77.90.31.21:4877 , piece_id=WTCYZ7PKCT27JZDXSLXQ7SPAPO5OBBOECVKRMULKB7WWJT7GXENA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.968831491 (289h39m5.839914536s) [noise=true , address=45.91.116.11:4117 , piece_id=BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.969248144 (289h39m5.840331189s) [noise=true , address=145.79.227.10:4402 , piece_id=S4K3TT37OZSDS4KDOC4YAALVXUTY2IHUKXQGS5R3OLSD6624S3MQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:07.095691332 (289h39m6.966774377s) [noise=true , address=77.90.31.21:4877 , piece_id=WTCYZ7PKCT27JZDXSLXQ7SPAPO5OBBOECVKRMULKB7WWJT7GXENA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 17:08:07.677933289 (289h39m7.549016334s) [upload=253937]: Next returned nil segment Dec 17 17:08:07.677950371 (289h39m7.549033406s) [upload=253937]: Waiting for error group managing segments... Dec 17 17:08:07.958796824 (289h39m7.829879869s) [noise=true , address=145.79.227.36:4517 , piece_id=7PKDM7LC3PDZ33DZY5AM3TQOI25WVGUFNAWL6TSOXQ5Y52VSTBRA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.967087221 (289h39m7.838170266s) [noise=true , address=145.79.227.10:4402 , piece_id=S4K3TT37OZSDS4KDOC4YAALVXUTY2IHUKXQGS5R3OLSD6624S3MQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.968141288 (289h39m7.839224333s) [noise=true , address=145.79.227.47:4266 , piece_id=BDPKINHG3BEUUYUMWK323J3HQOPOZBZNMACVMWDZ6URRGEG5ZV7A, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.974117077 (289h39m7.845200113s) [noise=true , address=145.79.227.46:4601 , piece_id=OQF64QD2T3F7Q2ZHYOF2PBY7T3SXBJE76Q4XVZFNPEDGMXPIVJ6Q, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.974188333 (289h39m7.845271378s) [noise=true , address=145.79.227.18:4006 , piece_id=ZX5SWPZWZSPEGEZB75LJGS6OQAEZQQPOUTAYG6HTADXXJBT75ZXQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.977968792 (289h39m7.849051837s) [noise=true , address=145.79.227.34:4907 , piece_id=D3NCY5V32N7AP23QEZYIU5GOKWCX5A4DMYMRYWTMSX3OJ6JETJNA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.980381085 (289h39m7.85146413s) [noise=true , address=145.79.227.13:4803 , piece_id=FUWPMH3PGYS64TDV2YZLARRYO2Q3MZTGJKCPCXLXGAOLZRFWNYTA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.026636303 (289h39m7.897719348s) [noise=true , address=145.79.227.43:4158 , piece_id=TURS2YMHYW23WZGMIYZ5YFVYFQYBKJKDZTRPMVR7U4APIAFZNAEA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.027057596 (289h39m7.898140641s) [noise=true , address=145.79.227.11:4730 , piece_id=ZVC3HDEOMANUTXQ4ORLFH3SDNJ3HRLVRVH6GHEKDAFYOZQK2OOYA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.046147687 (289h39m7.917230732s) [noise=true , address=145.79.227.48:4303 , piece_id=EH6WZNR7PU5B3DIVMN5LRHDIJXEU5P7SN4XHVER6JK74LYW5PM7Q, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.264192205 (289h39m8.13527525s) [noise=true , address=45.91.116.11:4117 , piece_id=BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.514532037 (289h39m8.385615083s) [noise=true , address=45.91.116.12:4226 , piece_id=DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.581684037 (289h39m8.452767082s) [noise=true , address=45.91.116.14:4190 , piece_id=AWTFBZTNXUM4N2UO7OFMENKYF7YMFTEIQWIIB2DZKBFSG5KTFBMQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.581708444 (289h39m8.452791489s) [seg_pos={0 0}, upload=253937]: Segment reached optimal threshold of 13 pieces. Dec 17 17:08:08.581998486 (289h39m8.453081521s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 17 17:08:08.582014847 (289h39m8.453097882s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Long tail cancellation due to optimal threshold Dec 17 17:08:08.582124086 (289h39m8.453207141s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 17 17:08:08.582133874 (289h39m8.453216909s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Long tail cancellation due to optimal threshold Dec 17 17:08:08.582169963 (289h39m8.453253008s) [seg_pos={0 0}, upload=253937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:08:08.582283388 (289h39m8.453366444s) [upload=253937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:08:08.582292756 (289h39m8.453375801s) [upload=253937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:08:08.873888509 (289h39m8.744971554s) [upload=253937]: Done uploading segments... Dec 17 17:08:08.873913225 (289h39m8.74499628s) [upload=253937]: Upload finished. err: <nil> Duration: 5.752569127s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 12.760783042s trace: https://jaeger.dev.storj.io/trace/21f11fd3717f95f0 FAIL (2025-12-17 17:13:45 UTC) orbiter warning: context deadline exceeded Duration: 12.760800375s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XXXXXXX.XX..XX.....X....X...XX..X..XX...X...................] duration: 5.001186124s trace: https://jaeger.dev.storj.io/trace/79887e0b0ab4d774 FAIL (2025-12-17 17:13:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00121086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X......................XX......] duration: 5.709438176s trace: https://jaeger.dev.storj.io/trace/435a94c4a7186daa FAIL (2025-12-17 17:13:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ) context deadline exceeded ======LOGS====== Dec 17 17:08:05.711119168 (289h39m5.582202214s) [upload=253937]: Starting upload Dec 17 17:08:05.711211945 (289h39m5.58229498s) [upload=253937]: Done starting upload Dec 17 17:08:05.711475638 (289h39m5.582558683s) [upload=253937]: Uploading segments... Dec 17 17:08:05.711534189 (289h39m5.582617244s) [upload=253937]: Got next segment. Inline: false Dec 17 17:08:05.711575557 (289h39m5.582658602s) [upload=253937]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 17 17:08:05.711581730 (289h39m5.582664775s) [upload=253937]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 17 17:08:05.961618917 (289h39m5.832701952s) [seg_pos={0 0}, upload=253937]: Begin upload segment... Dec 17 17:08:05.961679182 (289h39m5.832762227s) [seg_pos={0 0}, upload=253937]: Done begin upload segment. Dec 17 17:08:05.961761188 (289h39m5.832844243s) [noise=true , address=145.79.227.43:4158 , piece_id=TURS2YMHYW23WZGMIYZ5YFVYFQYBKJKDZTRPMVR7U4APIAFZNAEA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.961839697 (289h39m5.832922742s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.962527207 (289h39m5.833610252s) [noise=true , address=45.91.116.14:4190 , piece_id=AWTFBZTNXUM4N2UO7OFMENKYF7YMFTEIQWIIB2DZKBFSG5KTFBMQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.962964238 (289h39m5.834047283s) [noise=true , address=145.79.227.13:4803 , piece_id=FUWPMH3PGYS64TDV2YZLARRYO2Q3MZTGJKCPCXLXGAOLZRFWNYTA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.963481283 (289h39m5.834564338s) [noise=true , address=45.91.116.12:4226 , piece_id=DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.963977248 (289h39m5.835060293s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.964461880 (289h39m5.835544925s) [noise=true , address=145.79.227.34:4907 , piece_id=D3NCY5V32N7AP23QEZYIU5GOKWCX5A4DMYMRYWTMSX3OJ6JETJNA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.964959188 (289h39m5.836042233s) [noise=true , address=145.79.227.47:4266 , piece_id=BDPKINHG3BEUUYUMWK323J3HQOPOZBZNMACVMWDZ6URRGEG5ZV7A, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.965479038 (289h39m5.836562083s) [noise=true , address=145.79.227.36:4517 , piece_id=7PKDM7LC3PDZ33DZY5AM3TQOI25WVGUFNAWL6TSOXQ5Y52VSTBRA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.966053180 (289h39m5.837136225s) [noise=true , address=145.79.227.48:4303 , piece_id=EH6WZNR7PU5B3DIVMN5LRHDIJXEU5P7SN4XHVER6JK74LYW5PM7Q, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.966599952 (289h39m5.837682997s) [noise=true , address=145.79.227.46:4601 , piece_id=OQF64QD2T3F7Q2ZHYOF2PBY7T3SXBJE76Q4XVZFNPEDGMXPIVJ6Q, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.967138547 (289h39m5.838221582s) [noise=true , address=145.79.227.11:4730 , piece_id=ZVC3HDEOMANUTXQ4ORLFH3SDNJ3HRLVRVH6GHEKDAFYOZQK2OOYA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.967402069 (289h39m5.838485114s) [noise=true , address=145.79.227.18:4006 , piece_id=ZX5SWPZWZSPEGEZB75LJGS6OQAEZQQPOUTAYG6HTADXXJBT75ZXQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.968131428 (289h39m5.839214473s) [noise=true , address=77.90.31.21:4877 , piece_id=WTCYZ7PKCT27JZDXSLXQ7SPAPO5OBBOECVKRMULKB7WWJT7GXENA, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.968831491 (289h39m5.839914536s) [noise=true , address=45.91.116.11:4117 , piece_id=BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:05.969248144 (289h39m5.840331189s) [noise=true , address=145.79.227.10:4402 , piece_id=S4K3TT37OZSDS4KDOC4YAALVXUTY2IHUKXQGS5R3OLSD6624S3MQ, seg_pos={0 0}, upload=253937]: Uploading piece... Dec 17 17:08:07.095691332 (289h39m6.966774377s) [noise=true , address=77.90.31.21:4877 , piece_id=WTCYZ7PKCT27JZDXSLXQ7SPAPO5OBBOECVKRMULKB7WWJT7GXENA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 17 17:08:07.677933289 (289h39m7.549016334s) [upload=253937]: Next returned nil segment Dec 17 17:08:07.677950371 (289h39m7.549033406s) [upload=253937]: Waiting for error group managing segments... Dec 17 17:08:07.958796824 (289h39m7.829879869s) [noise=true , address=145.79.227.36:4517 , piece_id=7PKDM7LC3PDZ33DZY5AM3TQOI25WVGUFNAWL6TSOXQ5Y52VSTBRA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.967087221 (289h39m7.838170266s) [noise=true , address=145.79.227.10:4402 , piece_id=S4K3TT37OZSDS4KDOC4YAALVXUTY2IHUKXQGS5R3OLSD6624S3MQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.968141288 (289h39m7.839224333s) [noise=true , address=145.79.227.47:4266 , piece_id=BDPKINHG3BEUUYUMWK323J3HQOPOZBZNMACVMWDZ6URRGEG5ZV7A, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.974117077 (289h39m7.845200113s) [noise=true , address=145.79.227.46:4601 , piece_id=OQF64QD2T3F7Q2ZHYOF2PBY7T3SXBJE76Q4XVZFNPEDGMXPIVJ6Q, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.974188333 (289h39m7.845271378s) [noise=true , address=145.79.227.18:4006 , piece_id=ZX5SWPZWZSPEGEZB75LJGS6OQAEZQQPOUTAYG6HTADXXJBT75ZXQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.977968792 (289h39m7.849051837s) [noise=true , address=145.79.227.34:4907 , piece_id=D3NCY5V32N7AP23QEZYIU5GOKWCX5A4DMYMRYWTMSX3OJ6JETJNA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:07.980381085 (289h39m7.85146413s) [noise=true , address=145.79.227.13:4803 , piece_id=FUWPMH3PGYS64TDV2YZLARRYO2Q3MZTGJKCPCXLXGAOLZRFWNYTA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.026636303 (289h39m7.897719348s) [noise=true , address=145.79.227.43:4158 , piece_id=TURS2YMHYW23WZGMIYZ5YFVYFQYBKJKDZTRPMVR7U4APIAFZNAEA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.027057596 (289h39m7.898140641s) [noise=true , address=145.79.227.11:4730 , piece_id=ZVC3HDEOMANUTXQ4ORLFH3SDNJ3HRLVRVH6GHEKDAFYOZQK2OOYA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.046147687 (289h39m7.917230732s) [noise=true , address=145.79.227.48:4303 , piece_id=EH6WZNR7PU5B3DIVMN5LRHDIJXEU5P7SN4XHVER6JK74LYW5PM7Q, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.264192205 (289h39m8.13527525s) [noise=true , address=45.91.116.11:4117 , piece_id=BAFMFM3DEPE5PKNQXGA2ZZYXXCLEDOSLCE4XR4TXTQCPEEK67NCQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.514532037 (289h39m8.385615083s) [noise=true , address=45.91.116.12:4226 , piece_id=DYEN732V4BXVDXRGC4ZBALPOP57F2YOZKFDNTF2FYCEXV5YK2WGQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.581684037 (289h39m8.452767082s) [noise=true , address=45.91.116.14:4190 , piece_id=AWTFBZTNXUM4N2UO7OFMENKYF7YMFTEIQWIIB2DZKBFSG5KTFBMQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: <nil> Dec 17 17:08:08.581708444 (289h39m8.452791489s) [seg_pos={0 0}, upload=253937]: Segment reached optimal threshold of 13 pieces. Dec 17 17:08:08.581998486 (289h39m8.453081521s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 17 17:08:08.582014847 (289h39m8.453097882s) [noise=true , address=77.90.31.19:4844 , piece_id=RLHTWLEMNC7KLQWF5ZXTXLQ3HFKJIHSKMN5IPGH4BAB6TREPWUJQ, seg_pos={0 0}, upload=253937]: Long tail cancellation due to optimal threshold Dec 17 17:08:08.582124086 (289h39m8.453207141s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 17 17:08:08.582133874 (289h39m8.453216909s) [noise=true , address=145.79.227.44:4194 , piece_id=FJ5LOMLPQRAA3B2J7PVPSA5WTVKTNVHHVG25Z5P5ZUGI6OHGFHXA, seg_pos={0 0}, upload=253937]: Long tail cancellation due to optimal threshold Dec 17 17:08:08.582169963 (289h39m8.453253008s) [seg_pos={0 0}, upload=253937]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 17 17:08:08.582283388 (289h39m8.453366444s) [upload=253937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 17 17:08:08.582292756 (289h39m8.453375801s) [upload=253937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 17 17:08:08.873888509 (289h39m8.744971554s) [upload=253937]: Done uploading segments... Dec 17 17:08:08.873913225 (289h39m8.74499628s) [upload=253937]: Upload finished. err: <nil> Duration: 5.709464566s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)