Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X.X.........X..........X.....X......X............] duration: 5.751207417s FAIL (2025-08-12 17:31:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:20:42.856892460 (95h54m37.287972591s) [upload=99912]: Starting upload Aug 12 17:20:42.856971581 (95h54m37.288051702s) [upload=99912]: Done starting upload Aug 12 17:20:42.857123261 (95h54m37.288203393s) [upload=99912]: Uploading segments... Aug 12 17:20:42.857162626 (95h54m37.288242757s) [upload=99912]: Got next segment. Inline: false Aug 12 17:20:42.857186793 (95h54m37.288266914s) [upload=99912]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:20:42.857191311 (95h54m37.288271442s) [upload=99912]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:20:43.073720025 (95h54m37.504800156s) [upload=99912]: Next segment err: metaclient: Too Many Requests Aug 12 17:20:43.073740063 (95h54m37.504820194s) [upload=99912]: Done uploading segments... Aug 12 17:20:43.073751104 (95h54m37.504831225s) [upload=99912]: Upload finished. err: metaclient: Too Many Requests Duration: 5.751232615s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...........X...........] duration: 1.858105565s FAIL (2025-08-12 17:31:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:22:30.940149904 (95h56m25.371230035s) [upload=99941]: Starting upload Aug 12 17:22:30.940203987 (95h56m25.371284118s) [upload=99941]: Done starting upload Aug 12 17:22:30.940278659 (95h56m25.37135879s) [upload=99941]: Uploading segments... Aug 12 17:22:30.940327653 (95h56m25.371407784s) [upload=99941]: Got next segment. Inline: true Aug 12 17:22:30.940332051 (95h56m25.371412182s) [upload=99941]: Waiting for error group managing segments... Aug 12 17:22:30.940370956 (95h56m25.371451088s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:22:30.940376367 (95h56m25.371456498s) [upload=99941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:22:30.940379032 (95h56m25.371459163s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:22:31.175876729 (95h56m25.60695686s) [upload=99941]: Done uploading segments... Aug 12 17:22:31.175898940 (95h56m25.606979101s) [upload=99941]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.858133358s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.204036661s FAIL (2025-08-12 17:30:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 6.204064262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X.X.........X..........X.....X......X...........] duration: 5.932141595s FAIL (2025-08-12 17:30:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:20:42.856892460 (95h54m37.287972591s) [upload=99912]: Starting upload Aug 12 17:20:42.856971581 (95h54m37.288051702s) [upload=99912]: Done starting upload Aug 12 17:20:42.857123261 (95h54m37.288203393s) [upload=99912]: Uploading segments... Aug 12 17:20:42.857162626 (95h54m37.288242757s) [upload=99912]: Got next segment. Inline: false Aug 12 17:20:42.857186793 (95h54m37.288266914s) [upload=99912]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:20:42.857191311 (95h54m37.288271442s) [upload=99912]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:20:43.073720025 (95h54m37.504800156s) [upload=99912]: Next segment err: metaclient: Too Many Requests Aug 12 17:20:43.073740063 (95h54m37.504820194s) [upload=99912]: Done uploading segments... Aug 12 17:20:43.073751104 (95h54m37.504831225s) [upload=99912]: Upload finished. err: metaclient: Too Many Requests Duration: 5.932172685s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...........X..........] duration: 1.900046664s FAIL (2025-08-12 17:30:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:22:30.940149904 (95h56m25.371230035s) [upload=99941]: Starting upload Aug 12 17:22:30.940203987 (95h56m25.371284118s) [upload=99941]: Done starting upload Aug 12 17:22:30.940278659 (95h56m25.37135879s) [upload=99941]: Uploading segments... Aug 12 17:22:30.940327653 (95h56m25.371407784s) [upload=99941]: Got next segment. Inline: true Aug 12 17:22:30.940332051 (95h56m25.371412182s) [upload=99941]: Waiting for error group managing segments... Aug 12 17:22:30.940370956 (95h56m25.371451088s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:22:30.940376367 (95h56m25.371456498s) [upload=99941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:22:30.940379032 (95h56m25.371459163s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:22:31.175876729 (95h56m25.60695686s) [upload=99941]: Done uploading segments... Aug 12 17:22:31.175898940 (95h56m25.606979101s) [upload=99941]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.900101828s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.170522548s FAIL (2025-08-12 17:30:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 6.17054425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...........X.........] duration: 1.906554695s FAIL (2025-08-12 17:29:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:22:30.940149904 (95h56m25.371230035s) [upload=99941]: Starting upload Aug 12 17:22:30.940203987 (95h56m25.371284118s) [upload=99941]: Done starting upload Aug 12 17:22:30.940278659 (95h56m25.37135879s) [upload=99941]: Uploading segments... Aug 12 17:22:30.940327653 (95h56m25.371407784s) [upload=99941]: Got next segment. Inline: true Aug 12 17:22:30.940332051 (95h56m25.371412182s) [upload=99941]: Waiting for error group managing segments... Aug 12 17:22:30.940370956 (95h56m25.371451088s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:22:30.940376367 (95h56m25.371456498s) [upload=99941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:22:30.940379032 (95h56m25.371459163s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:22:31.175876729 (95h56m25.60695686s) [upload=99941]: Done uploading segments... Aug 12 17:22:31.175898940 (95h56m25.606979101s) [upload=99941]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.906573551s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X.X.........X..........X.....X......X..........] duration: 5.896491746s FAIL (2025-08-12 17:29:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:20:42.856892460 (95h54m37.287972591s) [upload=99912]: Starting upload Aug 12 17:20:42.856971581 (95h54m37.288051702s) [upload=99912]: Done starting upload Aug 12 17:20:42.857123261 (95h54m37.288203393s) [upload=99912]: Uploading segments... Aug 12 17:20:42.857162626 (95h54m37.288242757s) [upload=99912]: Got next segment. Inline: false Aug 12 17:20:42.857186793 (95h54m37.288266914s) [upload=99912]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:20:42.857191311 (95h54m37.288271442s) [upload=99912]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:20:43.073720025 (95h54m37.504800156s) [upload=99912]: Next segment err: metaclient: Too Many Requests Aug 12 17:20:43.073740063 (95h54m37.504820194s) [upload=99912]: Done uploading segments... Aug 12 17:20:43.073751104 (95h54m37.504831225s) [upload=99912]: Upload finished. err: metaclient: Too Many Requests Duration: 5.89652002s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.909725113s FAIL (2025-08-12 17:28:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 5.909753718s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...........X........] duration: 1.935928935s FAIL (2025-08-12 17:28:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:22:30.940149904 (95h56m25.371230035s) [upload=99941]: Starting upload Aug 12 17:22:30.940203987 (95h56m25.371284118s) [upload=99941]: Done starting upload Aug 12 17:22:30.940278659 (95h56m25.37135879s) [upload=99941]: Uploading segments... Aug 12 17:22:30.940327653 (95h56m25.371407784s) [upload=99941]: Got next segment. Inline: true Aug 12 17:22:30.940332051 (95h56m25.371412182s) [upload=99941]: Waiting for error group managing segments... Aug 12 17:22:30.940370956 (95h56m25.371451088s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:22:30.940376367 (95h56m25.371456498s) [upload=99941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:22:30.940379032 (95h56m25.371459163s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:22:31.175876729 (95h56m25.60695686s) [upload=99941]: Done uploading segments... Aug 12 17:22:31.175898940 (95h56m25.606979101s) [upload=99941]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.935954002s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X.X.........X..........X.....X......X.........] duration: 5.844572474s FAIL (2025-08-12 17:28:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:20:42.856892460 (95h54m37.287972591s) [upload=99912]: Starting upload Aug 12 17:20:42.856971581 (95h54m37.288051702s) [upload=99912]: Done starting upload Aug 12 17:20:42.857123261 (95h54m37.288203393s) [upload=99912]: Uploading segments... Aug 12 17:20:42.857162626 (95h54m37.288242757s) [upload=99912]: Got next segment. Inline: false Aug 12 17:20:42.857186793 (95h54m37.288266914s) [upload=99912]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:20:42.857191311 (95h54m37.288271442s) [upload=99912]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:20:43.073720025 (95h54m37.504800156s) [upload=99912]: Next segment err: metaclient: Too Many Requests Aug 12 17:20:43.073740063 (95h54m37.504820194s) [upload=99912]: Done uploading segments... Aug 12 17:20:43.073751104 (95h54m37.504831225s) [upload=99912]: Upload finished. err: metaclient: Too Many Requests Duration: 5.844595469s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...........X.......] duration: 1.855920354s FAIL (2025-08-12 17:27:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:22:30.940149904 (95h56m25.371230035s) [upload=99941]: Starting upload Aug 12 17:22:30.940203987 (95h56m25.371284118s) [upload=99941]: Done starting upload Aug 12 17:22:30.940278659 (95h56m25.37135879s) [upload=99941]: Uploading segments... Aug 12 17:22:30.940327653 (95h56m25.371407784s) [upload=99941]: Got next segment. Inline: true Aug 12 17:22:30.940332051 (95h56m25.371412182s) [upload=99941]: Waiting for error group managing segments... Aug 12 17:22:30.940370956 (95h56m25.371451088s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:22:30.940376367 (95h56m25.371456498s) [upload=99941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:22:30.940379032 (95h56m25.371459163s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:22:31.175876729 (95h56m25.60695686s) [upload=99941]: Done uploading segments... Aug 12 17:22:31.175898940 (95h56m25.606979101s) [upload=99941]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.855948388s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.016185829s FAIL (2025-08-12 17:27:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 6.016214193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............X.X.........X..........X.....X......X........] duration: 5.927599515s FAIL (2025-08-12 17:27:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:20:42.856892460 (95h54m37.287972591s) [upload=99912]: Starting upload Aug 12 17:20:42.856971581 (95h54m37.288051702s) [upload=99912]: Done starting upload Aug 12 17:20:42.857123261 (95h54m37.288203393s) [upload=99912]: Uploading segments... Aug 12 17:20:42.857162626 (95h54m37.288242757s) [upload=99912]: Got next segment. Inline: false Aug 12 17:20:42.857186793 (95h54m37.288266914s) [upload=99912]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:20:42.857191311 (95h54m37.288271442s) [upload=99912]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:20:43.073720025 (95h54m37.504800156s) [upload=99912]: Next segment err: metaclient: Too Many Requests Aug 12 17:20:43.073740063 (95h54m37.504820194s) [upload=99912]: Done uploading segments... Aug 12 17:20:43.073751104 (95h54m37.504831225s) [upload=99912]: Upload finished. err: metaclient: Too Many Requests Duration: 5.927625887s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...........X......] duration: 1.861889545s FAIL (2025-08-12 17:27:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:22:30.940149904 (95h56m25.371230035s) [upload=99941]: Starting upload Aug 12 17:22:30.940203987 (95h56m25.371284118s) [upload=99941]: Done starting upload Aug 12 17:22:30.940278659 (95h56m25.37135879s) [upload=99941]: Uploading segments... Aug 12 17:22:30.940327653 (95h56m25.371407784s) [upload=99941]: Got next segment. Inline: true Aug 12 17:22:30.940332051 (95h56m25.371412182s) [upload=99941]: Waiting for error group managing segments... Aug 12 17:22:30.940370956 (95h56m25.371451088s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:22:30.940376367 (95h56m25.371456498s) [upload=99941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:22:30.940379032 (95h56m25.371459163s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:22:31.175876729 (95h56m25.60695686s) [upload=99941]: Done uploading segments... Aug 12 17:22:31.175898940 (95h56m25.606979101s) [upload=99941]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.861914322s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.87905763s FAIL (2025-08-12 17:26:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 5.879085824s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [................X.X.........X..........X.....X......X.......] duration: 5.882419379s FAIL (2025-08-12 17:26:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:20:42.856892460 (95h54m37.287972591s) [upload=99912]: Starting upload Aug 12 17:20:42.856971581 (95h54m37.288051702s) [upload=99912]: Done starting upload Aug 12 17:20:42.857123261 (95h54m37.288203393s) [upload=99912]: Uploading segments... Aug 12 17:20:42.857162626 (95h54m37.288242757s) [upload=99912]: Got next segment. Inline: false Aug 12 17:20:42.857186793 (95h54m37.288266914s) [upload=99912]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:20:42.857191311 (95h54m37.288271442s) [upload=99912]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:20:43.073720025 (95h54m37.504800156s) [upload=99912]: Next segment err: metaclient: Too Many Requests Aug 12 17:20:43.073740063 (95h54m37.504820194s) [upload=99912]: Done uploading segments... Aug 12 17:20:43.073751104 (95h54m37.504831225s) [upload=99912]: Upload finished. err: metaclient: Too Many Requests Duration: 5.882442744s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...........X.....] duration: 1.866740216s FAIL (2025-08-12 17:26:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:22:30.940149904 (95h56m25.371230035s) [upload=99941]: Starting upload Aug 12 17:22:30.940203987 (95h56m25.371284118s) [upload=99941]: Done starting upload Aug 12 17:22:30.940278659 (95h56m25.37135879s) [upload=99941]: Uploading segments... Aug 12 17:22:30.940327653 (95h56m25.371407784s) [upload=99941]: Got next segment. Inline: true Aug 12 17:22:30.940332051 (95h56m25.371412182s) [upload=99941]: Waiting for error group managing segments... Aug 12 17:22:30.940370956 (95h56m25.371451088s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:22:30.940376367 (95h56m25.371456498s) [upload=99941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:22:30.940379032 (95h56m25.371459163s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:22:31.175876729 (95h56m25.60695686s) [upload=99941]: Done uploading segments... Aug 12 17:22:31.175898940 (95h56m25.606979101s) [upload=99941]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.866781224s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.036824992s FAIL (2025-08-12 17:26:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 6.036852975s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................X.X.........X..........X.....X......X......] duration: 5.948374113s FAIL (2025-08-12 17:26:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:20:42.856892460 (95h54m37.287972591s) [upload=99912]: Starting upload Aug 12 17:20:42.856971581 (95h54m37.288051702s) [upload=99912]: Done starting upload Aug 12 17:20:42.857123261 (95h54m37.288203393s) [upload=99912]: Uploading segments... Aug 12 17:20:42.857162626 (95h54m37.288242757s) [upload=99912]: Got next segment. Inline: false Aug 12 17:20:42.857186793 (95h54m37.288266914s) [upload=99912]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:20:42.857191311 (95h54m37.288271442s) [upload=99912]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:20:43.073720025 (95h54m37.504800156s) [upload=99912]: Next segment err: metaclient: Too Many Requests Aug 12 17:20:43.073740063 (95h54m37.504820194s) [upload=99912]: Done uploading segments... Aug 12 17:20:43.073751104 (95h54m37.504831225s) [upload=99912]: Upload finished. err: metaclient: Too Many Requests Duration: 5.94839903s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X.X.........X..........X.....X......X.....] duration: 6.001646862s FAIL (2025-08-12 17:25:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:20:42.856892460 (95h54m37.287972591s) [upload=99912]: Starting upload Aug 12 17:20:42.856971581 (95h54m37.288051702s) [upload=99912]: Done starting upload Aug 12 17:20:42.857123261 (95h54m37.288203393s) [upload=99912]: Uploading segments... Aug 12 17:20:42.857162626 (95h54m37.288242757s) [upload=99912]: Got next segment. Inline: false Aug 12 17:20:42.857186793 (95h54m37.288266914s) [upload=99912]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:20:42.857191311 (95h54m37.288271442s) [upload=99912]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:20:43.073720025 (95h54m37.504800156s) [upload=99912]: Next segment err: metaclient: Too Many Requests Aug 12 17:20:43.073740063 (95h54m37.504820194s) [upload=99912]: Done uploading segments... Aug 12 17:20:43.073751104 (95h54m37.504831225s) [upload=99912]: Upload finished. err: metaclient: Too Many Requests Duration: 6.001668973s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.103779614s FAIL (2025-08-12 17:25:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 6.103799021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...........X....] duration: 1.854512229s FAIL (2025-08-12 17:25:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:22:30.940149904 (95h56m25.371230035s) [upload=99941]: Starting upload Aug 12 17:22:30.940203987 (95h56m25.371284118s) [upload=99941]: Done starting upload Aug 12 17:22:30.940278659 (95h56m25.37135879s) [upload=99941]: Uploading segments... Aug 12 17:22:30.940327653 (95h56m25.371407784s) [upload=99941]: Got next segment. Inline: true Aug 12 17:22:30.940332051 (95h56m25.371412182s) [upload=99941]: Waiting for error group managing segments... Aug 12 17:22:30.940370956 (95h56m25.371451088s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:22:30.940376367 (95h56m25.371456498s) [upload=99941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:22:30.940379032 (95h56m25.371459163s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:22:31.175876729 (95h56m25.60695686s) [upload=99941]: Done uploading segments... Aug 12 17:22:31.175898940 (95h56m25.606979101s) [upload=99941]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.854540944s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [X...........................................X...........X...] duration: 1.929129539s FAIL (2025-08-12 17:24:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:22:30.940149904 (95h56m25.371230035s) [upload=99941]: Starting upload Aug 12 17:22:30.940203987 (95h56m25.371284118s) [upload=99941]: Done starting upload Aug 12 17:22:30.940278659 (95h56m25.37135879s) [upload=99941]: Uploading segments... Aug 12 17:22:30.940327653 (95h56m25.371407784s) [upload=99941]: Got next segment. Inline: true Aug 12 17:22:30.940332051 (95h56m25.371412182s) [upload=99941]: Waiting for error group managing segments... Aug 12 17:22:30.940370956 (95h56m25.371451088s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:22:30.940376367 (95h56m25.371456498s) [upload=99941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:22:30.940379032 (95h56m25.371459163s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:22:31.175876729 (95h56m25.60695686s) [upload=99941]: Done uploading segments... Aug 12 17:22:31.175898940 (95h56m25.606979101s) [upload=99941]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.929153064s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................X.X.........X..........X.....X......X....] duration: 5.923303783s FAIL (2025-08-12 17:24:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:20:42.856892460 (95h54m37.287972591s) [upload=99912]: Starting upload Aug 12 17:20:42.856971581 (95h54m37.288051702s) [upload=99912]: Done starting upload Aug 12 17:20:42.857123261 (95h54m37.288203393s) [upload=99912]: Uploading segments... Aug 12 17:20:42.857162626 (95h54m37.288242757s) [upload=99912]: Got next segment. Inline: false Aug 12 17:20:42.857186793 (95h54m37.288266914s) [upload=99912]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:20:42.857191311 (95h54m37.288271442s) [upload=99912]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:20:43.073720025 (95h54m37.504800156s) [upload=99912]: Next segment err: metaclient: Too Many Requests Aug 12 17:20:43.073740063 (95h54m37.504820194s) [upload=99912]: Done uploading segments... Aug 12 17:20:43.073751104 (95h54m37.504831225s) [upload=99912]: Upload finished. err: metaclient: Too Many Requests Duration: 5.923333821s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.803869117s FAIL (2025-08-12 17:24:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 5.803895638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X...........................................X...........X..] duration: 1.90882615s FAIL (2025-08-12 17:24:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:22:30.940149904 (95h56m25.371230035s) [upload=99941]: Starting upload Aug 12 17:22:30.940203987 (95h56m25.371284118s) [upload=99941]: Done starting upload Aug 12 17:22:30.940278659 (95h56m25.37135879s) [upload=99941]: Uploading segments... Aug 12 17:22:30.940327653 (95h56m25.371407784s) [upload=99941]: Got next segment. Inline: true Aug 12 17:22:30.940332051 (95h56m25.371412182s) [upload=99941]: Waiting for error group managing segments... Aug 12 17:22:30.940370956 (95h56m25.371451088s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:22:30.940376367 (95h56m25.371456498s) [upload=99941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:22:30.940379032 (95h56m25.371459163s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:22:31.175876729 (95h56m25.60695686s) [upload=99941]: Done uploading segments... Aug 12 17:22:31.175898940 (95h56m25.606979101s) [upload=99941]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.908856749s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................X.X.........X..........X.....X......X...] duration: 5.793782981s FAIL (2025-08-12 17:23:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:20:42.856892460 (95h54m37.287972591s) [upload=99912]: Starting upload Aug 12 17:20:42.856971581 (95h54m37.288051702s) [upload=99912]: Done starting upload Aug 12 17:20:42.857123261 (95h54m37.288203393s) [upload=99912]: Uploading segments... Aug 12 17:20:42.857162626 (95h54m37.288242757s) [upload=99912]: Got next segment. Inline: false Aug 12 17:20:42.857186793 (95h54m37.288266914s) [upload=99912]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:20:42.857191311 (95h54m37.288271442s) [upload=99912]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:20:43.073720025 (95h54m37.504800156s) [upload=99912]: Next segment err: metaclient: Too Many Requests Aug 12 17:20:43.073740063 (95h54m37.504820194s) [upload=99912]: Done uploading segments... Aug 12 17:20:43.073751104 (95h54m37.504831225s) [upload=99912]: Upload finished. err: metaclient: Too Many Requests Duration: 5.793809651s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.161031325s FAIL (2025-08-12 17:23:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 6.161076552s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X...........................................X...........X.] duration: 1.950473826s FAIL (2025-08-12 17:23:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:22:30.940149904 (95h56m25.371230035s) [upload=99941]: Starting upload Aug 12 17:22:30.940203987 (95h56m25.371284118s) [upload=99941]: Done starting upload Aug 12 17:22:30.940278659 (95h56m25.37135879s) [upload=99941]: Uploading segments... Aug 12 17:22:30.940327653 (95h56m25.371407784s) [upload=99941]: Got next segment. Inline: true Aug 12 17:22:30.940332051 (95h56m25.371412182s) [upload=99941]: Waiting for error group managing segments... Aug 12 17:22:30.940370956 (95h56m25.371451088s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:22:30.940376367 (95h56m25.371456498s) [upload=99941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:22:30.940379032 (95h56m25.371459163s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:22:31.175876729 (95h56m25.60695686s) [upload=99941]: Done uploading segments... Aug 12 17:22:31.175898940 (95h56m25.606979101s) [upload=99941]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.950494645s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.00729505s FAIL (2025-08-12 17:22:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 6.007318574s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...X...........................................X...........X] duration: 891.642593ms FAIL (2025-08-12 17:22:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:22:30.940149904 (95h56m25.371230035s) [upload=99941]: Starting upload Aug 12 17:22:30.940203987 (95h56m25.371284118s) [upload=99941]: Done starting upload Aug 12 17:22:30.940278659 (95h56m25.37135879s) [upload=99941]: Uploading segments... Aug 12 17:22:30.940327653 (95h56m25.371407784s) [upload=99941]: Got next segment. Inline: true Aug 12 17:22:30.940332051 (95h56m25.371412182s) [upload=99941]: Waiting for error group managing segments... Aug 12 17:22:30.940370956 (95h56m25.371451088s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:22:30.940376367 (95h56m25.371456498s) [upload=99941]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:22:30.940379032 (95h56m25.371459163s) [upload=99941]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:22:31.175876729 (95h56m25.60695686s) [upload=99941]: Done uploading segments... Aug 12 17:22:31.175898940 (95h56m25.606979101s) [upload=99941]: Upload finished. err: metaclient: Too Many Requests Duration:* 891.663863ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................X.X.........X..........X.....X......X..] duration: 6.014320299s FAIL (2025-08-12 17:22:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:20:42.856892460 (95h54m37.287972591s) [upload=99912]: Starting upload Aug 12 17:20:42.856971581 (95h54m37.288051702s) [upload=99912]: Done starting upload Aug 12 17:20:42.857123261 (95h54m37.288203393s) [upload=99912]: Uploading segments... Aug 12 17:20:42.857162626 (95h54m37.288242757s) [upload=99912]: Got next segment. Inline: false Aug 12 17:20:42.857186793 (95h54m37.288266914s) [upload=99912]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:20:42.857191311 (95h54m37.288271442s) [upload=99912]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:20:43.073720025 (95h54m37.504800156s) [upload=99912]: Next segment err: metaclient: Too Many Requests Aug 12 17:20:43.073740063 (95h54m37.504820194s) [upload=99912]: Done uploading segments... Aug 12 17:20:43.073751104 (95h54m37.504831225s) [upload=99912]: Upload finished. err: metaclient: Too Many Requests Duration: 6.014347961s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.813910567s FAIL (2025-08-12 17:22:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 5.813935285s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.099214754s FAIL (2025-08-12 17:21:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 6.099247707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X...........................................X...........] duration: 1.999503185s FAIL (2025-08-12 17:21:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 12 17:14:10.610708770 (95h48m5.041788891s) [upload=99796]: Starting upload Aug 12 17:14:10.610760419 (95h48m5.04184055s) [upload=99796]: Done starting upload Aug 12 17:14:10.610837096 (95h48m5.041917227s) [upload=99796]: Uploading segments... Aug 12 17:14:10.610918822 (95h48m5.041998954s) [upload=99796]: Got next segment. Inline: true Aug 12 17:14:10.610924463 (95h48m5.042004594s) [upload=99796]: Waiting for error group managing segments... Aug 12 17:14:10.610977775 (95h48m5.042057906s) [upload=99796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:14:10.610983366 (95h48m5.042063497s) [upload=99796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:14:10.610985930 (95h48m5.042066051s) [upload=99796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:14:10.896688638 (95h48m5.327768759s) [upload=99796]: Done uploading segments... Aug 12 17:14:10.896709046 (95h48m5.327789177s) [upload=99796]: Upload finished. err: <nil> Duration:* 1.999535577s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................X.X.........X..........X.....X......X.] duration: 6.131283913s FAIL (2025-08-12 17:21:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:20:42.856892460 (95h54m37.287972591s) [upload=99912]: Starting upload Aug 12 17:20:42.856971581 (95h54m37.288051702s) [upload=99912]: Done starting upload Aug 12 17:20:42.857123261 (95h54m37.288203393s) [upload=99912]: Uploading segments... Aug 12 17:20:42.857162626 (95h54m37.288242757s) [upload=99912]: Got next segment. Inline: false Aug 12 17:20:42.857186793 (95h54m37.288266914s) [upload=99912]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:20:42.857191311 (95h54m37.288271442s) [upload=99912]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:20:43.073720025 (95h54m37.504800156s) [upload=99912]: Next segment err: metaclient: Too Many Requests Aug 12 17:20:43.073740063 (95h54m37.504820194s) [upload=99912]: Done uploading segments... Aug 12 17:20:43.073751104 (95h54m37.504831225s) [upload=99912]: Upload finished. err: metaclient: Too Many Requests Duration: 6.131305796s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.941490009s FAIL (2025-08-12 17:20:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 5.941515237s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................X.X.........X..........X.....X......X] duration: 870.30547ms FAIL (2025-08-12 17:20:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 12 17:20:42.856892460 (95h54m37.287972591s) [upload=99912]: Starting upload Aug 12 17:20:42.856971581 (95h54m37.288051702s) [upload=99912]: Done starting upload Aug 12 17:20:42.857123261 (95h54m37.288203393s) [upload=99912]: Uploading segments... Aug 12 17:20:42.857162626 (95h54m37.288242757s) [upload=99912]: Got next segment. Inline: false Aug 12 17:20:42.857186793 (95h54m37.288266914s) [upload=99912]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:20:42.857191311 (95h54m37.288271442s) [upload=99912]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:20:43.073720025 (95h54m37.504800156s) [upload=99912]: Next segment err: metaclient: Too Many Requests Aug 12 17:20:43.073740063 (95h54m37.504820194s) [upload=99912]: Done uploading segments... Aug 12 17:20:43.073751104 (95h54m37.504831225s) [upload=99912]: Upload finished. err: metaclient: Too Many Requests Duration: 870.320759ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X...........................................X..........] duration: 2.176438883s FAIL (2025-08-12 17:20:31 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 12 17:14:10.610708770 (95h48m5.041788891s) [upload=99796]: Starting upload Aug 12 17:14:10.610760419 (95h48m5.04184055s) [upload=99796]: Done starting upload Aug 12 17:14:10.610837096 (95h48m5.041917227s) [upload=99796]: Uploading segments... Aug 12 17:14:10.610918822 (95h48m5.041998954s) [upload=99796]: Got next segment. Inline: true Aug 12 17:14:10.610924463 (95h48m5.042004594s) [upload=99796]: Waiting for error group managing segments... Aug 12 17:14:10.610977775 (95h48m5.042057906s) [upload=99796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:14:10.610983366 (95h48m5.042063497s) [upload=99796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:14:10.610985930 (95h48m5.042066051s) [upload=99796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:14:10.896688638 (95h48m5.327768759s) [upload=99796]: Done uploading segments... Aug 12 17:14:10.896709046 (95h48m5.327789177s) [upload=99796]: Upload finished. err: <nil> Duration:* 2.176465695s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.931081148s FAIL (2025-08-12 17:20:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 5.931110384s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................X.X.........X..........X.....X......] duration: 6.058532534s FAIL (2025-08-12 17:19:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 12 17:13:28.825252727 (95h47m23.256332898s) [upload=99787]: Starting upload Aug 12 17:13:28.825321419 (95h47m23.25640159s) [upload=99787]: Done starting upload Aug 12 17:13:28.825449604 (95h47m23.256529775s) [upload=99787]: Uploading segments... Aug 12 17:13:28.825744037 (95h47m23.256824198s) [upload=99787]: Got next segment. Inline: false Aug 12 17:13:28.825821235 (95h47m23.256901416s) [upload=99787]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:13:28.825829931 (95h47m23.256910093s) [upload=99787]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:13:29.303728856 (95h47m23.734808987s) [seg_pos={0 0}, upload=99787]: Begin upload segment... Aug 12 17:13:29.303773140 (95h47m23.734853271s) [seg_pos={0 0}, upload=99787]: Done begin upload segment. Aug 12 17:13:29.303793930 (95h47m23.734874061s) [noise=true , address=104.36.82.44:4900 , piece_id=LCDUXPUFOZ43NKZ4LOR5ZCVQ3RPPCTIR2UU7XDNNFHHXDS77UKFQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.303866910 (95h47m23.734947041s) [noise=true , address=104.36.82.58:4266 , piece_id=XOTBH7GZGN7QZFS2U2AS72TKR3AU2KYQD474Y6GMFFUYT4LHPLHQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.304176021 (95h47m23.735256152s) [noise=true , address=104.36.82.38:4694 , piece_id=RZ6LNKXDHDRU73N6NIULQG34PVWCYWIVA4FJ2BEF3CKUCFXGAKMQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.304419978 (95h47m23.735500109s) [noise=true , address=104.36.82.9:4025 , piece_id=AQJMXPICHPUI5UPVL6WZWMGZ7UFBVDK5KS4IFV3XBOXX76DMJVRA, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.304639759 (95h47m23.73571988s) [noise=true , address=79.127.219.51:4195 , piece_id=4EU53SFRSCZ2NNZBFWI427YQHR7BEFT5JX6TW7E33P2TRIN4FQXQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.304780888 (95h47m23.735861019s) [noise=true , address=64.49.13.154:4936 , piece_id=FF3TYKME6R42QWQEW27PSKJ5ORUC4A7SMWVHPH6L2ODX3TISQH5A, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.304926106 (95h47m23.736006227s) [noise=true , address=104.36.82.62:4441 , piece_id=MC7SVGPDMN4VVKEOB5CEERHLNI53KA6XGSWXE6OQWCMEL2XTWVJA, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.305191725 (95h47m23.736271856s) [noise=true , address=104.36.82.56:4237 , piece_id=BV33QX3IWIUX45JH7NJPW6A2BYF2CWF3PYVPC4YJQ75KCZ3JEXVA, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.305257680 (95h47m23.736337821s) [noise=true , address=64.49.13.140:4506 , piece_id=SSZ5XSTLM4CKZTRLABB2CC5BK54E3BGYX5HWWLXRNVK5BIPRVEFQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.305590016 (95h47m23.736670147s) [noise=true , address=104.36.82.19:4402 , piece_id=IEL4AM22HO26IC5TSAFKXCMHY3B4MGPOTDD2HVQPYEFD3GCJCADQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.305886914 (95h47m23.736967035s) [noise=true , address=104.36.82.17:4345 , piece_id=VVRYNEI7FMSK2QHNR7N2EACZQZVBKG54ADWESJNGS43PVEG5EMLQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.305961698 (95h47m23.737041829s) [noise=true , address=104.36.82.45:4931 , piece_id=WKX4RE7H2PMKMEZWTKT6EF2BALALCT2ZPNFAUFMC7IFUJITEJ7AQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.306106645 (95h47m23.737186776s) [noise=true , address=104.36.82.7:4915 , piece_id=35GNLEYYZL7J2RXDJ6ONZOX3SHEJHUCKVJDAGV64KWKURW5PCTNQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.306212778 (95h47m23.737292909s) [noise=true , address=104.36.82.11:4142 , piece_id=7D5FHBGX6S6CTMTK6IEUEYNOT73LQXJYZV6HMUVVP6OOPNR46R6Q, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.306306056 (95h47m23.737386187s) [noise=true , address=104.36.82.39:4736 , piece_id=77BJAQPGH6SJN3D3K5UPCA2DNG4T7SK5ECK6ZN2LSVK46EYRXAEQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.306459700 (95h47m23.737539832s) [noise=true , address=104.36.82.17:4344 , piece_id=PN5TP33O3LL7W5QJKB24GTSPNXBBKXIGETVIHSBRG4O252NY3T3A, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.306513974 (95h47m23.737594105s) [noise=true , address=104.36.82.60:4348 , piece_id=ARK7NJB5532ZZHNVMVYG3XCBOUT4JCO5CEBRC2365T66ERZSTJEA, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.306714267 (95h47m23.737794398s) [noise=true , address=77.90.31.16:4764 , piece_id=PVK7KVJD4ZTS2EA2FOU5APWROVZNYCZHGN3M4BXRHU4QJXGDGWWA, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:30.465351213 (95h47m24.896431344s) [upload=99787]: Next returned nil segment Aug 12 17:13:30.465368435 (95h47m24.896448567s) [upload=99787]: Waiting for error group managing segments... Aug 12 17:13:30.784923783 (95h47m25.216003914s) [noise=true , address=64.49.13.140:4506 , piece_id=SSZ5XSTLM4CKZTRLABB2CC5BK54E3BGYX5HWWLXRNVK5BIPRVEFQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:30.790033850 (95h47m25.221113981s) [noise=true , address=64.49.13.154:4936 , piece_id=FF3TYKME6R42QWQEW27PSKJ5ORUC4A7SMWVHPH6L2ODX3TISQH5A, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.255533719 (95h47m25.68661386s) [noise=true , address=104.36.82.38:4694 , piece_id=RZ6LNKXDHDRU73N6NIULQG34PVWCYWIVA4FJ2BEF3CKUCFXGAKMQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.269997042 (95h47m25.701077183s) [noise=true , address=104.36.82.60:4348 , piece_id=ARK7NJB5532ZZHNVMVYG3XCBOUT4JCO5CEBRC2365T66ERZSTJEA, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.302029471 (95h47m25.733109622s) [noise=true , address=77.90.31.16:4764 , piece_id=PVK7KVJD4ZTS2EA2FOU5APWROVZNYCZHGN3M4BXRHU4QJXGDGWWA, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.318353594 (95h47m25.749433725s) [noise=true , address=104.36.82.19:4402 , piece_id=IEL4AM22HO26IC5TSAFKXCMHY3B4MGPOTDD2HVQPYEFD3GCJCADQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.346017534 (95h47m25.777097665s) [noise=true , address=104.36.82.9:4025 , piece_id=AQJMXPICHPUI5UPVL6WZWMGZ7UFBVDK5KS4IFV3XBOXX76DMJVRA, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.348171306 (95h47m25.779251437s) [noise=true , address=104.36.82.45:4931 , piece_id=WKX4RE7H2PMKMEZWTKT6EF2BALALCT2ZPNFAUFMC7IFUJITEJ7AQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.354598163 (95h47m25.785678294s) [noise=true , address=104.36.82.56:4237 , piece_id=BV33QX3IWIUX45JH7NJPW6A2BYF2CWF3PYVPC4YJQ75KCZ3JEXVA, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.360073037 (95h47m25.791153169s) [noise=true , address=104.36.82.17:4345 , piece_id=VVRYNEI7FMSK2QHNR7N2EACZQZVBKG54ADWESJNGS43PVEG5EMLQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.371327111 (95h47m25.802407242s) [noise=true , address=104.36.82.39:4736 , piece_id=77BJAQPGH6SJN3D3K5UPCA2DNG4T7SK5ECK6ZN2LSVK46EYRXAEQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.372622479 (95h47m25.8037026s) [noise=true , address=104.36.82.44:4900 , piece_id=LCDUXPUFOZ43NKZ4LOR5ZCVQ3RPPCTIR2UU7XDNNFHHXDS77UKFQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.379217397 (95h47m25.810297528s) [noise=true , address=104.36.82.17:4344 , piece_id=PN5TP33O3LL7W5QJKB24GTSPNXBBKXIGETVIHSBRG4O252NY3T3A, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.393498702 (95h47m25.824578823s) [noise=true , address=104.36.82.58:4266 , piece_id=XOTBH7GZGN7QZFS2U2AS72TKR3AU2KYQD474Y6GMFFUYT4LHPLHQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.393527026 (95h47m25.824607157s) [seg_pos={0 0}, upload=99787]: Segment reached optimal threshold of 14 pieces. Aug 12 17:13:31.393647137 (95h47m25.824727268s) [noise=true , address=104.36.82.11:4142 , piece_id=7D5FHBGX6S6CTMTK6IEUEYNOT73LQXJYZV6HMUVVP6OOPNR46R6Q, seg_pos={0 0}, upload=99787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7): protocol: expected piece hash; context canceled; EOF Aug 12 17:13:31.393705218 (95h47m25.824785339s) [noise=true , address=79.127.219.51:4195 , piece_id=4EU53SFRSCZ2NNZBFWI427YQHR7BEFT5JX6TW7E33P2TRIN4FQXQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mFXuncCmi6nsnuVqGK9QxfGCzGXgX3kcPPy9icg1fPemHpsx): protocol: expected piece hash; context canceled; EOF Aug 12 17:13:31.393756705 (95h47m25.824836836s) [noise=true , address=104.36.82.62:4441 , piece_id=MC7SVGPDMN4VVKEOB5CEERHLNI53KA6XGSWXE6OQWCMEL2XTWVJA, seg_pos={0 0}, upload=99787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VBayWisMK1n3cwypeiLzGuffonBTzYga7thAZ8pSqrusc7kow): protocol: expected piece hash; context canceled; EOF Aug 12 17:13:31.393778618 (95h47m25.824858739s) [noise=true , address=104.36.82.7:4915 , piece_id=35GNLEYYZL7J2RXDJ6ONZOX3SHEJHUCKVJDAGV64KWKURW5PCTNQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt): protocol: expected piece hash; context canceled; EOF Aug 12 17:13:31.393833052 (95h47m25.824913183s) [seg_pos={0 0}, upload=99787]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 12 17:13:31.393916311 (95h47m25.824996442s) [upload=99787]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 17:13:31.393920689 (95h47m25.82500082s) [upload=99787]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:13:31.701564897 (95h47m26.132645038s) [upload=99787]: Done uploading segments... Aug 12 17:13:31.701588933 (95h47m26.132669114s) [upload=99787]: Upload finished. err: <nil> Duration: 6.058556339s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X...........................................X.........] duration: 1.867535297s FAIL (2025-08-12 17:19:45 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 12 17:14:10.610708770 (95h48m5.041788891s) [upload=99796]: Starting upload Aug 12 17:14:10.610760419 (95h48m5.04184055s) [upload=99796]: Done starting upload Aug 12 17:14:10.610837096 (95h48m5.041917227s) [upload=99796]: Uploading segments... Aug 12 17:14:10.610918822 (95h48m5.041998954s) [upload=99796]: Got next segment. Inline: true Aug 12 17:14:10.610924463 (95h48m5.042004594s) [upload=99796]: Waiting for error group managing segments... Aug 12 17:14:10.610977775 (95h48m5.042057906s) [upload=99796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:14:10.610983366 (95h48m5.042063497s) [upload=99796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:14:10.610985930 (95h48m5.042066051s) [upload=99796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:14:10.896688638 (95h48m5.327768759s) [upload=99796]: Done uploading segments... Aug 12 17:14:10.896709046 (95h48m5.327789177s) [upload=99796]: Upload finished. err: <nil> Duration:* 1.867560585s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.23515032s FAIL (2025-08-12 17:19:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 6.235174557s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X...........................................X........] duration: 1.985189051s FAIL (2025-08-12 17:19:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 12 17:14:10.610708770 (95h48m5.041788891s) [upload=99796]: Starting upload Aug 12 17:14:10.610760419 (95h48m5.04184055s) [upload=99796]: Done starting upload Aug 12 17:14:10.610837096 (95h48m5.041917227s) [upload=99796]: Uploading segments... Aug 12 17:14:10.610918822 (95h48m5.041998954s) [upload=99796]: Got next segment. Inline: true Aug 12 17:14:10.610924463 (95h48m5.042004594s) [upload=99796]: Waiting for error group managing segments... Aug 12 17:14:10.610977775 (95h48m5.042057906s) [upload=99796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:14:10.610983366 (95h48m5.042063497s) [upload=99796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:14:10.610985930 (95h48m5.042066051s) [upload=99796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:14:10.896688638 (95h48m5.327768759s) [upload=99796]: Done uploading segments... Aug 12 17:14:10.896709046 (95h48m5.327789177s) [upload=99796]: Upload finished. err: <nil> Duration:* 1.98521395s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X...........................................X.......] duration: 2.639685661s FAIL (2025-08-12 17:18:45 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 12 17:14:10.610708770 (95h48m5.041788891s) [upload=99796]: Starting upload Aug 12 17:14:10.610760419 (95h48m5.04184055s) [upload=99796]: Done starting upload Aug 12 17:14:10.610837096 (95h48m5.041917227s) [upload=99796]: Uploading segments... Aug 12 17:14:10.610918822 (95h48m5.041998954s) [upload=99796]: Got next segment. Inline: true Aug 12 17:14:10.610924463 (95h48m5.042004594s) [upload=99796]: Waiting for error group managing segments... Aug 12 17:14:10.610977775 (95h48m5.042057906s) [upload=99796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:14:10.610983366 (95h48m5.042063497s) [upload=99796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:14:10.610985930 (95h48m5.042066051s) [upload=99796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:14:10.896688638 (95h48m5.327768759s) [upload=99796]: Done uploading segments... Aug 12 17:14:10.896709046 (95h48m5.327789177s) [upload=99796]: Upload finished. err: <nil> Duration:* 2.63971653s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................X.X.........X..........X.....X.....] duration: 5.967958032s FAIL (2025-08-12 17:18:44 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 12 17:13:28.825252727 (95h47m23.256332898s) [upload=99787]: Starting upload Aug 12 17:13:28.825321419 (95h47m23.25640159s) [upload=99787]: Done starting upload Aug 12 17:13:28.825449604 (95h47m23.256529775s) [upload=99787]: Uploading segments... Aug 12 17:13:28.825744037 (95h47m23.256824198s) [upload=99787]: Got next segment. Inline: false Aug 12 17:13:28.825821235 (95h47m23.256901416s) [upload=99787]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:13:28.825829931 (95h47m23.256910093s) [upload=99787]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 17:13:29.303728856 (95h47m23.734808987s) [seg_pos={0 0}, upload=99787]: Begin upload segment... Aug 12 17:13:29.303773140 (95h47m23.734853271s) [seg_pos={0 0}, upload=99787]: Done begin upload segment. Aug 12 17:13:29.303793930 (95h47m23.734874061s) [noise=true , address=104.36.82.44:4900 , piece_id=LCDUXPUFOZ43NKZ4LOR5ZCVQ3RPPCTIR2UU7XDNNFHHXDS77UKFQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.303866910 (95h47m23.734947041s) [noise=true , address=104.36.82.58:4266 , piece_id=XOTBH7GZGN7QZFS2U2AS72TKR3AU2KYQD474Y6GMFFUYT4LHPLHQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.304176021 (95h47m23.735256152s) [noise=true , address=104.36.82.38:4694 , piece_id=RZ6LNKXDHDRU73N6NIULQG34PVWCYWIVA4FJ2BEF3CKUCFXGAKMQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.304419978 (95h47m23.735500109s) [noise=true , address=104.36.82.9:4025 , piece_id=AQJMXPICHPUI5UPVL6WZWMGZ7UFBVDK5KS4IFV3XBOXX76DMJVRA, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.304639759 (95h47m23.73571988s) [noise=true , address=79.127.219.51:4195 , piece_id=4EU53SFRSCZ2NNZBFWI427YQHR7BEFT5JX6TW7E33P2TRIN4FQXQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.304780888 (95h47m23.735861019s) [noise=true , address=64.49.13.154:4936 , piece_id=FF3TYKME6R42QWQEW27PSKJ5ORUC4A7SMWVHPH6L2ODX3TISQH5A, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.304926106 (95h47m23.736006227s) [noise=true , address=104.36.82.62:4441 , piece_id=MC7SVGPDMN4VVKEOB5CEERHLNI53KA6XGSWXE6OQWCMEL2XTWVJA, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.305191725 (95h47m23.736271856s) [noise=true , address=104.36.82.56:4237 , piece_id=BV33QX3IWIUX45JH7NJPW6A2BYF2CWF3PYVPC4YJQ75KCZ3JEXVA, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.305257680 (95h47m23.736337821s) [noise=true , address=64.49.13.140:4506 , piece_id=SSZ5XSTLM4CKZTRLABB2CC5BK54E3BGYX5HWWLXRNVK5BIPRVEFQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.305590016 (95h47m23.736670147s) [noise=true , address=104.36.82.19:4402 , piece_id=IEL4AM22HO26IC5TSAFKXCMHY3B4MGPOTDD2HVQPYEFD3GCJCADQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.305886914 (95h47m23.736967035s) [noise=true , address=104.36.82.17:4345 , piece_id=VVRYNEI7FMSK2QHNR7N2EACZQZVBKG54ADWESJNGS43PVEG5EMLQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.305961698 (95h47m23.737041829s) [noise=true , address=104.36.82.45:4931 , piece_id=WKX4RE7H2PMKMEZWTKT6EF2BALALCT2ZPNFAUFMC7IFUJITEJ7AQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.306106645 (95h47m23.737186776s) [noise=true , address=104.36.82.7:4915 , piece_id=35GNLEYYZL7J2RXDJ6ONZOX3SHEJHUCKVJDAGV64KWKURW5PCTNQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.306212778 (95h47m23.737292909s) [noise=true , address=104.36.82.11:4142 , piece_id=7D5FHBGX6S6CTMTK6IEUEYNOT73LQXJYZV6HMUVVP6OOPNR46R6Q, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.306306056 (95h47m23.737386187s) [noise=true , address=104.36.82.39:4736 , piece_id=77BJAQPGH6SJN3D3K5UPCA2DNG4T7SK5ECK6ZN2LSVK46EYRXAEQ, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.306459700 (95h47m23.737539832s) [noise=true , address=104.36.82.17:4344 , piece_id=PN5TP33O3LL7W5QJKB24GTSPNXBBKXIGETVIHSBRG4O252NY3T3A, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.306513974 (95h47m23.737594105s) [noise=true , address=104.36.82.60:4348 , piece_id=ARK7NJB5532ZZHNVMVYG3XCBOUT4JCO5CEBRC2365T66ERZSTJEA, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:29.306714267 (95h47m23.737794398s) [noise=true , address=77.90.31.16:4764 , piece_id=PVK7KVJD4ZTS2EA2FOU5APWROVZNYCZHGN3M4BXRHU4QJXGDGWWA, seg_pos={0 0}, upload=99787]: Uploading piece... Aug 12 17:13:30.465351213 (95h47m24.896431344s) [upload=99787]: Next returned nil segment Aug 12 17:13:30.465368435 (95h47m24.896448567s) [upload=99787]: Waiting for error group managing segments... Aug 12 17:13:30.784923783 (95h47m25.216003914s) [noise=true , address=64.49.13.140:4506 , piece_id=SSZ5XSTLM4CKZTRLABB2CC5BK54E3BGYX5HWWLXRNVK5BIPRVEFQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:30.790033850 (95h47m25.221113981s) [noise=true , address=64.49.13.154:4936 , piece_id=FF3TYKME6R42QWQEW27PSKJ5ORUC4A7SMWVHPH6L2ODX3TISQH5A, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.255533719 (95h47m25.68661386s) [noise=true , address=104.36.82.38:4694 , piece_id=RZ6LNKXDHDRU73N6NIULQG34PVWCYWIVA4FJ2BEF3CKUCFXGAKMQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.269997042 (95h47m25.701077183s) [noise=true , address=104.36.82.60:4348 , piece_id=ARK7NJB5532ZZHNVMVYG3XCBOUT4JCO5CEBRC2365T66ERZSTJEA, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.302029471 (95h47m25.733109622s) [noise=true , address=77.90.31.16:4764 , piece_id=PVK7KVJD4ZTS2EA2FOU5APWROVZNYCZHGN3M4BXRHU4QJXGDGWWA, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.318353594 (95h47m25.749433725s) [noise=true , address=104.36.82.19:4402 , piece_id=IEL4AM22HO26IC5TSAFKXCMHY3B4MGPOTDD2HVQPYEFD3GCJCADQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.346017534 (95h47m25.777097665s) [noise=true , address=104.36.82.9:4025 , piece_id=AQJMXPICHPUI5UPVL6WZWMGZ7UFBVDK5KS4IFV3XBOXX76DMJVRA, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.348171306 (95h47m25.779251437s) [noise=true , address=104.36.82.45:4931 , piece_id=WKX4RE7H2PMKMEZWTKT6EF2BALALCT2ZPNFAUFMC7IFUJITEJ7AQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.354598163 (95h47m25.785678294s) [noise=true , address=104.36.82.56:4237 , piece_id=BV33QX3IWIUX45JH7NJPW6A2BYF2CWF3PYVPC4YJQ75KCZ3JEXVA, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.360073037 (95h47m25.791153169s) [noise=true , address=104.36.82.17:4345 , piece_id=VVRYNEI7FMSK2QHNR7N2EACZQZVBKG54ADWESJNGS43PVEG5EMLQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.371327111 (95h47m25.802407242s) [noise=true , address=104.36.82.39:4736 , piece_id=77BJAQPGH6SJN3D3K5UPCA2DNG4T7SK5ECK6ZN2LSVK46EYRXAEQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.372622479 (95h47m25.8037026s) [noise=true , address=104.36.82.44:4900 , piece_id=LCDUXPUFOZ43NKZ4LOR5ZCVQ3RPPCTIR2UU7XDNNFHHXDS77UKFQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.379217397 (95h47m25.810297528s) [noise=true , address=104.36.82.17:4344 , piece_id=PN5TP33O3LL7W5QJKB24GTSPNXBBKXIGETVIHSBRG4O252NY3T3A, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.393498702 (95h47m25.824578823s) [noise=true , address=104.36.82.58:4266 , piece_id=XOTBH7GZGN7QZFS2U2AS72TKR3AU2KYQD474Y6GMFFUYT4LHPLHQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: <nil> Aug 12 17:13:31.393527026 (95h47m25.824607157s) [seg_pos={0 0}, upload=99787]: Segment reached optimal threshold of 14 pieces. Aug 12 17:13:31.393647137 (95h47m25.824727268s) [noise=true , address=104.36.82.11:4142 , piece_id=7D5FHBGX6S6CTMTK6IEUEYNOT73LQXJYZV6HMUVVP6OOPNR46R6Q, seg_pos={0 0}, upload=99787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JR1vFqWgTH1qwvxH4DEk4eKT7TfxWC2Re6QEWmqxWXctytfa7): protocol: expected piece hash; context canceled; EOF Aug 12 17:13:31.393705218 (95h47m25.824785339s) [noise=true , address=79.127.219.51:4195 , piece_id=4EU53SFRSCZ2NNZBFWI427YQHR7BEFT5JX6TW7E33P2TRIN4FQXQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mFXuncCmi6nsnuVqGK9QxfGCzGXgX3kcPPy9icg1fPemHpsx): protocol: expected piece hash; context canceled; EOF Aug 12 17:13:31.393756705 (95h47m25.824836836s) [noise=true , address=104.36.82.62:4441 , piece_id=MC7SVGPDMN4VVKEOB5CEERHLNI53KA6XGSWXE6OQWCMEL2XTWVJA, seg_pos={0 0}, upload=99787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VBayWisMK1n3cwypeiLzGuffonBTzYga7thAZ8pSqrusc7kow): protocol: expected piece hash; context canceled; EOF Aug 12 17:13:31.393778618 (95h47m25.824858739s) [noise=true , address=104.36.82.7:4915 , piece_id=35GNLEYYZL7J2RXDJ6ONZOX3SHEJHUCKVJDAGV64KWKURW5PCTNQ, seg_pos={0 0}, upload=99787]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt): protocol: expected piece hash; context canceled; EOF Aug 12 17:13:31.393833052 (95h47m25.824913183s) [seg_pos={0 0}, upload=99787]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 12 17:13:31.393916311 (95h47m25.824996442s) [upload=99787]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 17:13:31.393920689 (95h47m25.82500082s) [upload=99787]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:13:31.701564897 (95h47m26.132645038s) [upload=99787]: Done uploading segments... Aug 12 17:13:31.701588933 (95h47m26.132669114s) [upload=99787]: Upload finished. err: <nil> Duration: 5.967984452s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.868269824s FAIL (2025-08-12 17:18:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 5.868295041s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.071836418s FAIL (2025-08-12 17:18:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 12 16:51:00.342944596 (95h24m54.774024727s) [upload=99416]: Starting upload Aug 12 16:51:00.343003058 (95h24m54.774083189s) [upload=99416]: Done starting upload Aug 12 16:51:00.343093682 (95h24m54.774173803s) [upload=99416]: Uploading segments... Aug 12 16:51:00.343214823 (95h24m54.774294954s) [upload=99416]: Got next segment. Inline: false Aug 12 16:51:00.343269027 (95h24m54.774349158s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 16:51:00.343273756 (95h24m54.774353887s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 12 16:51:00.585349151 (95h24m55.016429282s) [seg_pos={0 0}, upload=99416]: Begin upload segment... Aug 12 16:51:00.585396822 (95h24m55.016476973s) [seg_pos={0 0}, upload=99416]: Done begin upload segment. Aug 12 16:51:00.585410067 (95h24m55.016490208s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585507734 (95h24m55.016587895s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.585886159 (95h24m55.0169663s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586089607 (95h24m55.017169738s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.586541652 (95h24m55.017621803s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587004688 (95h24m55.018084819s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587619234 (95h24m55.018699385s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.587983842 (95h24m55.019063983s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588303153 (95h24m55.019383284s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.588688090 (95h24m55.019768221s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.589902393 (95h24m55.020982534s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.590674450 (95h24m55.021754601s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.591296961 (95h24m55.022377092s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.592189969 (95h24m55.02327011s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.593156619 (95h24m55.02423678s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:00.594056290 (95h24m55.025136421s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Uploading piece... Aug 12 16:51:02.362133336 (95h24m56.793213467s) [upload=99416]: Next returned nil segment Aug 12 16:51:02.362159547 (95h24m56.793239678s) [upload=99416]: Waiting for error group managing segments... Aug 12 16:51:02.650293798 (95h24m57.081373929s) [noise=true , address=77.90.31.18:4805 , piece_id=NW27AB2XE452YUWKJOSPZXQQ3O5THGJQZWPFNCYYTTD5KSM4LUIQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.663772758 (95h24m57.094852919s) [noise=true , address=77.90.31.12:4592 , piece_id=D5DIBKFCLXZCN5WGTVASSOPLLGWXBY42VRDRT4GERYQEN7TE7OQQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.696344749 (95h24m57.12742488s) [noise=true , address=77.90.31.17:4772 , piece_id=F335TOCUX6EOQEL3B6CRYVSNT3K3VPQKUIO4OQCKLUOYDPITGCRQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.709104884 (95h24m57.140185025s) [noise=true , address=104.36.82.12:4150 , piece_id=2URQGYBRQFL4GRLQTARGPS5OHFKAIMQ5LK4TBQJKEHWATWJQUIYA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.722233484 (95h24m57.153313625s) [noise=true , address=45.91.116.5:4299 , piece_id=KKS2MUX4QFJSJLZB6FQ7QUSIXNDXG5V666IDIXCZTVLIARPB5OOQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.727048907 (95h24m57.158129038s) [noise=true , address=77.90.31.16:4734 , piece_id=UDY6JVB3TPS2HOLPTX4ILBUJXC6IQC3BOZNVHKANKWELCBZ4O24A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.741870376 (95h24m57.172950507s) [noise=true , address=45.91.116.7:4009 , piece_id=74F32XPA5Z3DQY36ATNSVLFQSIAHA6EHL3T5W2HO2547GFJXWG7A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.772732365 (95h24m57.203812507s) [noise=true , address=104.36.82.14:4222 , piece_id=ZNU7RTT5A4ZIX23ERO2NKXX4QVNKLJB4R3VJRZVQ5SRRI5OWGC2A, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.805755481 (95h24m57.236835612s) [noise=true , address=104.36.82.4:4801 , piece_id=AZNJYD5EDG7BFS35RJP3MLEJCMPM6J7UKQIBA2RIXTXARTJ4FXGQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.878280266 (95h24m57.309360397s) [noise=true , address=77.90.31.10:4520 , piece_id=LQC7NVJGKRVXL34UQO7TDTRE3VAKZ5CENKPKM7JUSW4I4TQ3MZCQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.913734542 (95h24m57.344814673s) [noise=true , address=45.91.116.6:4975 , piece_id=MO7KK43ZBCOIDIQGWFRHNFWG5WZL77ZESIW4U24DXCC5BB57R52Q, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.972369120 (95h24m57.403449251s) [noise=true , address=104.36.82.10:4079 , piece_id=URAOYRC7A52G7O5TOAEPHJHT3G6GBDJZN3GTA37BQ5TBRSW4OYFQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983815812 (95h24m57.414895943s) [noise=true , address=104.36.82.2:4732 , piece_id=N4DXFGAMG72X2ER6AY53WWXX5KKUPAVCOVFWAFQ57KEHHFUXE2GQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: <nil> Aug 12 16:51:02.983842733 (95h24m57.414922864s) [seg_pos={0 0}, upload=99416]: Segment reached optimal threshold of 13 pieces. Aug 12 16:51:02.983942084 (95h24m57.415022215s) [noise=true , address=77.90.31.11:4554 , piece_id=HEXD7AOVOJTK37LL4SCH43UN4VN7EI7G7WFSPOPGHIGBPKTGV3DA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.983999583 (95h24m57.415079715s) [noise=true , address=45.91.116.7:4010 , piece_id=4UT6Y2N3KF4HO6YHIBQVF46Q5JNHLDJYBLLCFXFDOURAH4EVMBKA, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984087211 (95h24m57.415167342s) [noise=true , address=45.91.116.5:4298 , piece_id=FMRPFGWGVYKSC67IBWY5UFDVPN7KKRNOWZ5K66XRL4BSOXM6PVKQ, seg_pos={0 0}, upload=99416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Aug 12 16:51:02.984124363 (95h24m57.415204494s) [seg_pos={0 0}, upload=99416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 12 16:51:02.984229683 (95h24m57.415309814s) [upload=99416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 12 16:51:02.984234323 (95h24m57.415314454s) [upload=99416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 16:51:03.289516404 (95h24m57.720596575s) [upload=99416]: Done uploading segments... Aug 12 16:51:03.289540049 (95h24m57.7206202s) [upload=99416]: Upload finished. err: <nil> Duration: 6.071867437s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X...........................................X......] duration: 1.988006388s FAIL (2025-08-12 17:17:45 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 12 17:14:10.610708770 (95h48m5.041788891s) [upload=99796]: Starting upload Aug 12 17:14:10.610760419 (95h48m5.04184055s) [upload=99796]: Done starting upload Aug 12 17:14:10.610837096 (95h48m5.041917227s) [upload=99796]: Uploading segments... Aug 12 17:14:10.610918822 (95h48m5.041998954s) [upload=99796]: Got next segment. Inline: true Aug 12 17:14:10.610924463 (95h48m5.042004594s) [upload=99796]: Waiting for error group managing segments... Aug 12 17:14:10.610977775 (95h48m5.042057906s) [upload=99796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 12 17:14:10.610983366 (95h48m5.042063497s) [upload=99796]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 12 17:14:10.610985930 (95h48m5.042066051s) [upload=99796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 12 17:14:10.896688638 (95h48m5.327768759s) [upload=99796]: Done uploading segments... Aug 12 17:14:10.896709046 (95h48m5.327789177s) [upload=99796]: Upload finished. err: <nil> Duration:* 1.988033851s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)